Unified

Utility

Maximization

Framework

for

Resource

Selection

This

paper

presents

a

unified

utility

framework

for

resource

selection

of

distributed

text

information

retrieval

This

new

framework

shows

an

efficient

and

effective

way

to

infer

the

probabilities

of

relevance

of

all

the

documents

across

the

text

databases

With

the

estimated

relevance

information

resource

selection

can

be

made

by

explicitly

optimizing

the

goals

of

different

applications

Specifically

when

used

for

database

recommendation

the

selection

is

optimized

for

the

goal

of

high-recall

-LRB-

include

as

many

relevant

documents

as

possible

in

the

selected

databases

-RRB-

when

used

for

distributed

document

retrieval

the

selection

targets

the

high-precision

goal

-LRB-

high

precision

in

the

final

merged

list

of

documents

-RRB-

This

new

model

provides

a

more

solid

framework

for

distributed

information

retrieval

Empirical

studies

show

that

it

is

at

least

as

effective

as

other

state-of-the-art

algorithms

H.

-LSB-

Information

Search

and

Retrieval

-RSB-

Algorithms

Conventional

search

engines

such

as

Google

or

AltaVista

use

ad-hoc

information

retrieval

solution

by

assuming

all

the

searchable

documents

can

be

copied

into

a

single

centralized

database

for

the

purpose

of

indexing

Distributed

information

retrieval

also

known

as

federated

search

-LSB-

-RSB-

is

different

from

ad-hoc

information

retrieval

as

it

addresses

the

cases

when

documents

can

not

be

acquired

and

stored

in

a

single

database

For

example

Hidden

Web

contents

-LRB-

also

called

invisible

or

deep

Web

contents

-RRB-

are

information

on

the

Web

that

can

not

be

accessed

by

the

conventional

search

engines

Hidden

web

contents

have

been

estimated

to

be

-LSB-

-RSB-

times

larger

than

the

contents

that

can

be

searched

by

conventional

search

engines

Therefore

it

is

very

important

to

search

this

type

of

valuable

information

The

architecture

of

distributed

search

solution

is

highly

influenced

by

different

environmental

characteristics

In

a

small

local

area

network

such

as

small

company

environments

the

information

providers

may

cooperate

to

provide

corpus

statistics

or

use

the

same

type

of

search

engines

Early

distributed

information

retrieval

research

focused

on

this

type

of

cooperative

environments

-LSB-

-RSB-

On

the

other

side

in

a

wide

area

network

such

as

very

large

corporate

environments

or

on

the

Web

there

are

many

types

of

search

engines

and

it

is

difficult

to

assume

that

all

the

information

providers

can

cooperate

as

they

are

required

Even

if

they

are

willing

to

cooperate

in

these

environments

it

may

be

hard

to

enforce

a

single

solution

for

all

the

information

providers

or

to

detect

whether

information

sources

provide

the

correct

information

as

they

are

required

Many

applications

fall

into

the

latter

type

of

uncooperative

environments

such

as

the

Mind

project

-LSB-

-RSB-

which

integrates

non-cooperating

digital

libraries

or

the

QProber

system

-LSB-

-RSB-

which

supports

browsing

and

searching

of

uncooperative

hidden

Web

databases

In

this

paper

we

focus

mainly

on

uncooperative

environments

that

contain

multiple

types

of

independent

search

engines

There

are

three

important

sub-problems

in

distributed

information

retrieval

First

information

about

the

contents

of

each

individual

database

must

be

acquired

-LRB-

resource

representation

-RRB-

-LSB-

-RSB-

Second

given

a

query

a

set

of

resources

must

be

selected

to

do

the

search

-LRB-

resource

selection

-RRB-

-LSB-

-RSB-

Third

the

results

retrieved

from

all

the

selected

resources

have

to

be

merged

into

a

single

final

list

before

it

can

be

presented

to

the

end

user

-LRB-

retrieval

and

results

merging

-RRB-

-LSB-

-RSB-

Many

types

of

solutions

exist

for

distributed

information

retrieval

Invisible-web.net

provides

guided

browsing

of

hidden

Web

databases

by

collecting

the

resource

descriptions

of

these

databases

and

building

hierarchies

of

classes

that

group

them

by

similar

topics

A

database

recommendation

system

goes

a

step

further

than

a

browsing

system

like

Invisible-web.net

by

recommending

most

relevant

information

sources

to

users

queries

It

is

composed

of

the

resource

description

and

the

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

CIKM

November

Washington

DC

USA

Copyright

ACM

resource

selection

components

This

solution

is

useful

when

the

users

want

to

browse

the

selected

databases

by

themselves

instead

of

asking

the

system

to

retrieve

relevant

documents

automatically

Distributed

document

retrieval

is

a

more

sophisticated

task

It

selects

relevant

information

sources

for

users

queries

as

the

database

recommendation

system

does

Furthermore

users

queries

are

forwarded

to

the

corresponding

selected

databases

and

the

returned

individual

ranked

lists

are

merged

into

a

single

list

to

present

to

the

users

The

goal

of

a

database

recommendation

system

is

to

select

a

small

set

of

resources

that

contain

as

many

relevant

documents

as

possible

which

we

call

a

high-recall

goal

On

the

other

side

the

effectiveness

of

distributed

document

retrieval

is

often

measured

by

the

Precision

of

the

final

merged

document

result

list

which

we

call

a

high-precision

goal

Prior

research

indicated

that

these

two

goals

are

related

but

not

identical

-LSB-

-RSB-

However

most

previous

solutions

simply

use

effective

resource

selection

algorithm

of

database

recommendation

system

for

distributed

document

retrieval

system

or

solve

the

inconsistency

with

heuristic

methods

-LSB-

-RSB-

This

paper

presents

a

unified

utility

maximization

framework

to

integrate

the

resource

selection

problem

of

both

database

recommendation

and

distributed

document

retrieval

together

by

treating

them

as

different

optimization

goals

First

a

centralized

sample

database

is

built

by

randomly

sampling

a

small

amount

of

documents

from

each

database

with

query-based

sampling

-LSB-

-RSB-

database

size

statistics

are

also

estimated

-LSB-

-RSB-

A

logistic

transformation

model

is

learned

off

line

with

a

small

amount

of

training

queries

to

map

the

centralized

document

scores

in

the

centralized

sample

database

to

the

corresponding

probabilities

of

relevance

Second

after

a

new

query

is

submitted

the

query

can

be

used

to

search

the

centralized

sample

database

which

produces

a

score

for

each

sampled

document

The

probability

of

relevance

for

each

document

in

the

centralized

sample

database

can

be

estimated

by

applying

the

logistic

model

to

each

document

's

score

Then

the

probabilities

of

relevance

of

all

the

-LRB-

mostly

unseen

-RRB-

documents

among

the

available

databases

can

be

estimated

using

the

probabilities

of

relevance

of

the

documents

in

the

centralized

sample

database

and

the

database

size

estimates

For

the

task

of

resource

selection

for

a

database

recommendation

system

the

databases

can

be

ranked

by

the

expected

number

of

relevant

documents

to

meet

the

high-recall

goal

For

resource

selection

for

a

distributed

document

retrieval

system

databases

containing

a

small

number

of

documents

with

large

probabilities

of

relevance

are

favored

over

databases

containing

many

documents

with

small

probabilities

of

relevance

This

selection

criterion

meets

the

high-precision

goal

of

distributed

document

retrieval

application

Furthermore

the

Semi-supervised

learning

-LRB-

SSL

-RRB-

-LSB-

-RSB-

algorithm

is

applied

to

merge

the

returned

documents

into

a

final

ranked

list

The

unified

utility

framework

makes

very

few

assumptions

and

works

in

uncooperative

environments

Two

key

features

make

it

a

more

solid

model

for

distributed

information

retrieval

i

-RRB-

It

formalizes

the

resource

selection

problems

of

different

applications

as

various

utility

functions

and

optimizes

the

utility

functions

to

achieve

the

optimal

results

accordingly

and

ii

-RRB-

It

shows

an

effective

and

efficient

way

to

estimate

the

probabilities

of

relevance

of

all

documents

across

databases

Specifically

the

framework

builds

logistic

models

on

the

centralized

sample

database

to

transform

centralized

retrieval

scores

to

the

corresponding

probabilities

of

relevance

and

uses

the

centralized

sample

database

as

the

bridge

between

individual

databases

and

the

logistic

model

The

human

effort

-LRB-

relevance

judgment

-RRB-

required

to

train

the

single

centralized

logistic

model

does

not

scale

with

the

number

of

databases

This

is

a

large

advantage

over

previous

research

which

required

the

amount

of

human

effort

to

be

linear

with

the

number

of

databases

-LSB-

-RSB-

The

unified

utility

framework

is

not

only

more

theoretically

solid

but

also

very

effective

Empirical

studies

show

the

new

model

to

be

at

least

as

accurate

as

the

state-of-the-art

algorithms

in

a

variety

of

configurations

The

next

section

discusses

related

work

Section

describes

the

new

unified

utility

maximization

model

Section

explains

our

experimental

methodology

Sections

and

present

our

experimental

results

for

resource

selection

and

document

retrieval

Section

concludes

There

has

been

considerable

research

on

all

the

sub-problems

of

distributed

information

retrieval

We

survey

the

most

related

work

in

this

section

The

first

problem

of

distributed

information

retrieval

is

resource

representation

The

STARTS

protocol

is

one

solution

for

acquiring

resource

descriptions

in

cooperative

environments

-LSB-

-RSB-

However

in

uncooperative

environments

even

the

databases

are

willing

to

share

their

information

it

is

not

easy

to

judge

whether

the

information

they

provide

is

accurate

or

not

Furthermore

it

is

not

easy

to

coordinate

the

databases

to

provide

resource

representations

that

are

compatible

with

each

other

Thus

in

uncooperative

environments

one

common

choice

is

query-based

sampling

which

randomly

generates

and

sends

queries

to

individual

search

engines

and

retrieves

some

documents

to

build

the

descriptions

As

the

sampled

documents

are

selected

by

random

queries

query-based

sampling

is

not

easily

fooled

by

any

adversarial

spammer

that

is

interested

to

attract

more

traffic

Experiments

have

shown

that

rather

accurate

resource

descriptions

can

be

built

by

sending

about

queries

and

downloading

about

documents

-LSB-

-RSB-

Many

resource

selection

algorithms

such

as

gGlOSS/vGlOSS

-LSB-

-RSB-

and

CORI

-LSB-

-RSB-

have

been

proposed

in

the

last

decade

The

CORI

algorithm

represents

each

database

by

its

terms

the

document

frequencies

and

a

small

number

of

corpus

statistics

-LRB-

details

in

-LSB-

-RSB-

-RRB-

As

prior

research

on

different

datasets

has

shown

the

CORI

algorithm

to

be

the

most

stable

and

effective

of

the

three

algorithms

-LSB-

-RSB-

we

use

it

as

a

baseline

algorithm

in

this

work

The

relevant

document

distribution

estimation

-LRB-

ReDDE

-LSB-

-RSB-

-RRB-

resource

selection

algorithm

is

a

recent

algorithm

that

tries

to

estimate

the

distribution

of

relevant

documents

across

the

available

databases

and

ranks

the

databases

accordingly

Although

the

ReDDE

algorithm

has

been

shown

to

be

effective

it

relies

on

heuristic

constants

that

are

set

empirically

-LSB-

-RSB-

The

last

step

of

the

document

retrieval

sub-problem

is

results

merging

which

is

the

process

of

transforming

database-specific

document

scores

into

comparable

database-independent

document

scores

The

semi

supervised

learning

-LRB-

SSL

-RRB-

-LSB-

-RSB-

result

merging

algorithm

uses

the

documents

acquired

by

query-based

sampling

as

training

data

and

linear

regression

to

learn

the

database-specific

query-specific

merging

models

These

linear

models

are

used

to

convert

the

database-specific

document

scores

into

the

approximated

centralized

document

scores

The

SSL

algorithm

has

been

shown

to

be

effective

-LSB-

-RSB-

It

serves

as

an

important

component

of

our

unified

utility

maximization

framework

-LRB-

Section

-RRB-

In

order

to

achieve

accurate

document

retrieval

results

many

previous

methods

simply

use

resource

selection

algorithms

that

are

effective

of

database

recommendation

system

But

as

pointed

out

above

a

good

resource

selection

algorithm

optimized

for

high-recall

may

not

work

well

for

document

retrieval

which

targets

the

high-precision

goal

This

type

of

inconsistency

has

been

observed

in

previous

research

-LSB-

-RSB-

The

research

in

-LSB-

-RSB-

tried

to

solve

the

problem

with

a

heuristic

method

The

research

most

similar

to

what

we

propose

here

is

the

decision-theoretic

framework

-LRB-

DTF

-RRB-

-LSB-

-RSB-

This

framework

computes

a

selection

that

minimizes

the

overall

costs

-LRB-

e.g.

retrieval

quality

time

-RRB-

of

document

retrieval

system

and

several

methods

-LSB-

-RSB-

have

been

proposed

to

estimate

the

retrieval

quality

However

two

points

distinguish

our

research

from

the

DTF

model

First

the

DTF

is

a

framework

designed

specifically

for

document

retrieval

but

our

new

model

integrates

two

distinct

applications

with

different

requirements

-LRB-

database

recommendation

and

distributed

document

retrieval

-RRB-

into

the

same

unified

framework

Second

the

DTF

builds

a

model

for

each

database

to

calculate

the

probabilities

of

relevance

This

requires

human

relevance

judgments

for

the

results

retrieved

from

each

database

In

contrast

our

approach

only

builds

one

logistic

model

for

the

centralized

sample

database

The

centralized

sample

database

can

serve

as

a

bridge

to

connect

the

individual

databases

with

the

centralized

logistic

model

thus

the

probabilities

of

relevance

of

documents

in

different

databases

can

be

estimated

This

strategy

can

save

large

amount

of

human

judgment

effort

and

is

a

big

advantage

of

the

unified

utility

maximization

framework

over

the

DTF

especially

when

there

are

a

large

number

of

databases

The

Unified

Utility

Maximization

-LRB-

UUM

-RRB-

framework

is

based

on

estimating

the

probabilities

of

relevance

of

the

-LRB-

mostly

unseen

-RRB-

documents

available

in

the

distributed

search

environment

In

this

section

we

describe

how

the

probabilities

of

relevance

are

estimated

and

how

they

are

used

by

the

Unified

Utility

Maximization

model

We

also

describe

how

the

model

can

be

optimized

for

the

high-recall

goal

of

a

database

recommendation

system

and

the

high-precision

goal

of

a

distributed

document

retrieval

system

Estimating

Probabilities

of

Relevance

As

pointed

out

above

the

purpose

of

resource

selection

is

high-recall

and

the

purpose

of

document

retrieval

is

high-precision

In

order

to

meet

these

diverse

goals

the

key

issue

is

to

estimate

the

probabilities

of

relevance

of

the

documents

in

various

databases

This

is

a

difficult

problem

because

we

can

only

observe

a

sample

of

the

contents

of

each

database

using

query-based

sampling

Our

strategy

is

to

make

full

use

of

all

the

available

information

to

calculate

the

probability

estimates

Learning

Probabilities

of

Relevance

In

the

resource

description

step

the

centralized

sample

database

is

built

by

query-based

sampling

and

the

database

sizes

are

estimated

using

the

sample-resample

method

-LSB-

-RSB-

At

the

same

time

an

effective

retrieval

algorithm

-LRB-

Inquery

-LSB-

-RSB-

-RRB-

is

applied

on

the

centralized

sample

database

with

a

small

number

-LRB-

e.g.

-RRB-

of

training

queries

For

each

training

query

the

CORI

resource

selection

algorithm

-LSB-

-RSB-

is

applied

to

select

some

number

-LRB-

e.g.

-RRB-

of

databases

and

retrieve

document

ids

from

each

database

The

SSL

results

merging

algorithm

-LSB-

-RSB-

is

used

to

merge

the

results

Then

we

can

download

the

top

documents

in

the

final

merged

list

and

calculate

their

corresponding

centralized

scores

using

Inquery

and

the

corpus

statistics

of

the

centralized

sample

database

The

centralized

scores

are

further

normalized

-LRB-

divided

by

the

maximum

centralized

score

for

each

query

-RRB-

as

this

method

has

been

suggested

to

improve

estimation

accuracy

in

previous

research

-LSB-

-RSB-

Human

judgment

is

acquired

for

those

documents

and

a

logistic

model

is

built

to

transform

the

normalized

centralized

document

scores

to

probabilities

of

relevance

as

follows

-LRB-

-RRB-

-RRB-

-RRB-

-LRB-

exp

-LRB-

-RRB-

-RRB-

-LRB-

exp

-LRB-

-RRB-

-LRB-

d

S

b

a

d

S

b

a

d

rel

P

d

R

c

c

c

c

c

c

-LRB-

-RRB-

where

-RRB-

-LRB-

d

S

c

is

the

normalized

centralized

document

score

and

a

c

and

b

c

are

the

two

parameters

of

the

logistic

model

These

two

parameters

are

estimated

by

maximizing

the

probabilities

of

relevance

of

the

training

queries

The

logistic

model

provides

us

the

tool

to

calculate

the

probabilities

of

relevance

from

centralized

document

scores

Estimating

Centralized

Document

Scores

When

the

user

submits

a

new

query

the

centralized

document

scores

of

the

documents

in

the

centralized

sample

database

are

calculated

However

in

order

to

calculate

the

probabilities

of

relevance

we

need

to

estimate

centralized

document

scores

for

all

documents

across

the

databases

instead

of

only

the

sampled

documents

This

goal

is

accomplished

using

the

centralized

scores

of

the

documents

in

the

centralized

sample

database

and

the

database

size

statistics

We

define

the

database

scale

factor

for

the

i

th

database

as

the

ratio

of

the

estimated

database

size

and

the

number

of

documents

sampled

from

this

database

as

follows

i

i

i

db

db

db

samp

N

SF

N

-LRB-

-RRB-

where

i

db

N

is

the

estimated

database

size

and

i

db

samp

N

is

the

number

of

documents

from

the

i

th

database

in

the

centralized

sample

database

The

intuition

behind

the

database

scale

factor

is

that

for

a

database

whose

scale

factor

is

if

one

document

from

this

database

in

the

centralized

sample

database

has

a

centralized

document

score

of

we

may

guess

that

there

are

about

documents

in

that

database

which

have

scores

of

about

Actually

we

can

apply

a

finer

non-parametric

linear

interpolation

method

to

estimate

the

centralized

document

score

curve

for

each

database

Formally

we

rank

all

the

sampled

documents

from

the

i

th

database

by

their

centralized

document

scores

to

get

the

sampled

centralized

document

score

list

-LCB-

S

c

-LRB-

ds

-RRB-

S

c

-LRB-

ds

-RRB-

S

c

-LRB-

ds

-RRB-

-RCB-

for

the

i

th

database

we

assume

that

if

we

could

calculate

the

centralized

document

scores

for

all

the

documents

in

this

database

and

get

the

complete

centralized

document

score

list

the

top

document

in

the

sampled

list

would

have

rank

SF

dbi

the

second

document

in

the

sampled

list

would

rank

SF

dbi

and

so

on

Therefore

the

data

points

of

sampled

documents

in

the

complete

list

are

-LCB-

-LRB-

SF

dbi

S

c

-LRB-

ds

-RRB-

-RRB-

-LRB-

SF

dbi

S

c

-LRB-

ds

-RRB-

-RRB-

-LRB-

SF

dbi

S

c

-LRB-

ds

-RRB-

-RRB-

-RCB-

Piecewise

linear

interpolation

is

applied

to

estimate

the

centralized

document

score

curve

as

illustrated

in

Figure

The

complete

centralized

document

score

list

can

be

estimated

by

calculating

the

values

of

different

ranks

on

the

centralized

document

curve

as

-RSB-

-LSB-

-RRB-

-LRB-

S

c

i

db

ij

N

j

d

It

can

be

seen

from

Figure

that

more

sample

data

points

produce

more

accurate

estimates

of

the

centralized

document

score

curves

However

for

databases

with

large

database

scale

ratios

this

kind

of

linear

interpolation

may

be

rather

inaccurate

especially

for

the

top

ranked

-LRB-

e.g.

-LSB-

SF

dbi

-RSB-

-RRB-

documents

Therefore

an

alternative

solution

is

proposed

to

estimate

the

centralized

document

scores

of

the

top

ranked

documents

for

databases

with

large

scale

ratios

-LRB-

e.g.

larger

than

-RRB-

Specifically

a

logistic

model

is

built

for

each

of

these

databases

The

logistic

model

is

used

to

estimate

the

centralized

document

score

of

the

top

document

in

the

corresponding

database

by

using

the

two

sampled

documents

from

that

database

with

highest

centralized

scores

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

exp

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

exp

-LRB-

-RRB-

-LRB-

i

c

i

i

c

i

i

i

c

i

i

c

i

i

i

c

ds

S

ds

S

ds

S

ds

S

d

S

-LRB-

-RRB-

i

i

and

i

are

the

parameters

of

the

logistic

model

For

each

training

query

the

top

retrieved

document

of

each

database

is

downloaded

and

the

corresponding

centralized

document

score

is

calculated

Together

with

the

scores

of

the

top

two

sampled

documents

these

parameters

can

be

estimated

After

the

centralized

score

of

the

top

document

is

estimated

an

exponential

function

is

fitted

for

the

top

part

-LRB-

-LSB-

SF

dbi

-RSB-

-RRB-

of

the

centralized

document

score

curve

as

-RSB-

-LSB-

-RRB-

exp

-LRB-

-RRB-

-LRB-

i

db

i

i

ij

c

SF

j

j

d

S

-LRB-

-RRB-

log

-LRB-

-LRB-

-RRB-

-RRB-

i

c

i

i

S

d

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-RRB-

-LRB-

log

-LRB-

-RRB-

-LRB-

-LRB-

log

-LRB-

i

db

i

c

i

c

i

SF

d

S

ds

S

-LRB-

-RRB-

The

two

parameters

i

and

i

are

fitted

to

make

sure

the

exponential

function

passes

through

the

two

points

-LRB-

-RRB-

-LRB-

i

c

d

S

-RRB-

and

-LRB-

SF

dbi

S

c

-LRB-

ds

-RRB-

-RRB-

The

exponential

function

is

only

used

to

adjust

the

top

part

of

the

centralized

document

score

curve

and

the

lower

part

of

the

curve

is

still

fitted

with

the

linear

interpolation

method

described

above

The

adjustment

by

fitting

exponential

function

of

the

top

ranked

documents

has

been

shown

empirically

to

produce

more

accurate

results

From

the

centralized

document

score

curves

we

can

estimate

the

complete

centralized

document

score

lists

accordingly

for

all

the

available

databases

After

the

estimated

centralized

document

scores

are

normalized

the

complete

lists

of

probabilities

of

relevance

can

be

constructed

out

of

the

complete

centralized

document

score

lists

by

Equation

Formally

for

the

i

th

database

the

complete

list

of

probabilities

of

relevance

is

-RSB-

-LSB-

-RRB-

-LRB-

R

i

db

ij

N

j

d

The

Unified

Utility

Maximization

Model

In

this

section

we

formally

define

the

new

unified

utility

maximization

model

which

optimizes

the

resource

selection

problems

for

two

goals

of

high-recall

-LRB-

database

recommendation

-RRB-

and

high-precision

-LRB-

distributed

document

retrieval

-RRB-

in

the

same

framework

In

the

task

of

database

recommendation

the

system

needs

to

decide

how

to

rank

databases

In

the

task

of

document

retrieval

the

system

not

only

needs

to

select

the

databases

but

also

needs

to

decide

how

many

documents

to

retrieve

from

each

selected

database

We

generalize

the

database

recommendation

selection

process

which

implicitly

recommends

all

documents

in

every

selected

database

as

a

special

case

of

the

selection

decision

for

the

document

retrieval

task

Formally

we

denote

d

i

as

the

number

of

documents

we

would

like

to

retrieve

from

the

i

th

database

and

-RCB-

-LCB-

d

d

d

as

a

selection

action

for

all

the

databases

The

database

selection

decision

is

made

based

on

the

complete

lists

of

probabilities

of

relevance

for

all

the

databases

The

complete

lists

of

probabilities

of

relevance

are

inferred

from

all

the

available

information

specifically

s

R

which

stands

for

the

resource

descriptions

acquired

by

query-based

sampling

and

the

database

size

estimates

acquired

by

sample-resample

c

S

stands

for

the

centralized

document

scores

of

the

documents

in

the

centralized

sample

database

If

the

method

of

estimating

centralized

document

scores

and

probabilities

of

relevance

in

Section

is

acceptable

then

the

most

probable

complete

lists

of

probabilities

of

relevance

can

be

derived

and

we

denote

them

as

-LCB-

-LRB-

R

-LRB-

-RRB-

-LSB-

-RSB-

-RRB-

db

j

d

j

N

-LRB-

R

-LRB-

-RRB-

-LSB-

-RSB-

-RRB-

-RCB-

db

j

d

j

N

Random

vector

denotes

an

arbitrary

set

of

complete

lists

of

probabilities

of

relevance

and

-RRB-

-LRB-

c

s

S

R

P

as

the

probability

of

generating

this

set

of

lists

Finally

to

each

selection

action

d

and

a

set

of

complete

lists

of

Figure

Linear

interpolation

construction

of

the

complete

centralized

document

score

list

-LRB-

database

scale

factor

is

-RRB-

probabilities

of

relevance

we

associate

a

utility

function

-RRB-

-LRB-

d

U

which

indicates

the

benefit

from

making

the

d

selection

when

the

true

complete

lists

of

probabilities

of

relevance

are

Therefore

the

selection

decision

defined

by

the

Bayesian

framework

is

d

S

R

P

d

U

d

c

s

d

-RRB-

-LRB-

-RRB-

-LRB-

max

arg

-LRB-

-RRB-

One

common

approach

to

simplify

the

computation

in

the

Bayesian

framework

is

to

only

calculate

the

utility

function

at

the

most

probable

parameter

values

instead

of

calculating

the

whole

expectation

In

other

words

we

only

need

to

calculate

-RRB-

-LRB-

d

U

and

Equation

is

simplified

as

follows

-RRB-

-LRB-

max

arg

d

U

d

d

-LRB-

-RRB-

This

equation

serves

as

the

basic

model

for

both

the

database

recommendation

system

and

the

document

retrieval

system

Resource

Selection

for

High-Recall

High-recall

is

the

goal

of

the

resource

selection

algorithm

in

federated

search

tasks

such

as

database

recommendation

The

goal

is

to

select

a

small

set

of

resources

-LRB-

e.g.

less

than

N

sdb

databases

-RRB-

that

contain

as

many

relevant

documents

as

possible

which

can

be

formally

defined

as

i

N

j

ij

i

i

db

d

d

I

d

U

-RRB-

-LRB-

R

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

I

-LRB-

d

i

-RRB-

is

the

indicator

function

which

is

when

the

i

th

database

is

selected

and

otherwise

Plug

this

equation

into

the

basic

model

in

Equation

and

associate

the

selected

database

number

constraint

to

obtain

the

following

sdb

i

i

i

N

j

ij

i

d

N

d

I

to

Subject

d

d

I

d

i

db

-RRB-

-LRB-

-RRB-

-LRB-

R

-RRB-

-LRB-

max

arg

-LRB-

-RRB-

The

solution

of

this

optimization

problem

is

very

simple

We

can

calculate

the

expected

number

of

relevant

documents

for

each

database

as

follows

i

db

i

N

j

ij

Rd

d

N

-RRB-

-LRB-

R

-LRB-

-RRB-

The

N

sdb

databases

with

the

largest

expected

number

of

relevant

documents

can

be

selected

to

meet

the

high-recall

goal

We

call

this

the

UUM/HR

algorithm

-LRB-

Unified

Utility

Maximization

for

High-Recall

-RRB-

Resource

Selection

for

High-Precision

High-Precision

is

the

goal

of

resource

selection

algorithm

in

federated

search

tasks

such

as

distributed

document

retrieval

It

is

measured

by

the

Precision

at

the

top

part

of

the

final

merged

document

list

This

high-precision

criterion

is

realized

by

the

following

utility

function

which

measures

the

Precision

of

retrieved

documents

from

the

selected

databases

i

d

j

ij

i

i

d

d

I

d

U

-RRB-

-LRB-

R

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

Note

that

the

key

difference

between

Equation

and

Equation

is

that

Equation

sums

up

the

probabilities

of

relevance

of

all

the

documents

in

a

database

while

Equation

only

considers

a

much

smaller

part

of

the

ranking

Specifically

we

can

calculate

the

optimal

selection

decision

by

i

d

j

ij

i

d

i

d

d

I

d

-RRB-

-LRB-

R

-RRB-

-LRB-

max

arg

-LRB-

-RRB-

Different

kinds

of

constraints

caused

by

different

characteristics

of

the

document

retrieval

tasks

can

be

associated

with

the

above

optimization

problem

The

most

common

one

is

to

select

a

fixed

number

-LRB-

N

sdb

-RRB-

of

databases

and

retrieve

a

fixed

number

-LRB-

N

rdoc

-RRB-

of

documents

from

each

selected

database

formally

defined

as

-RRB-

-LRB-

-RRB-

-LRB-

R

-RRB-

-LRB-

max

arg

i

rdoc

i

sdb

i

i

i

d

j

ij

i

d

d

if

N

d

N

d

I

to

Subject

d

d

I

d

i

-LRB-

-RRB-

This

optimization

problem

can

be

solved

easily

by

calculating

the

number

of

expected

relevant

documents

in

the

top

part

of

the

each

database

's

complete

list

of

probabilities

of

relevance

rdoc

i

N

j

ij

Rd

Top

d

N

-RRB-

-LRB-

R

-LRB-

-RRB-

Then

the

databases

can

be

ranked

by

these

values

and

selected

We

call

this

the

UUM/HP-FL

algorithm

-LRB-

Unified

Utility

Maximization

for

High-Precision

with

Fixed

Length

document

rankings

from

each

selected

database

-RRB-

A

more

complex

situation

is

to

vary

the

number

of

retrieved

documents

from

each

selected

database

More

specifically

we

allow

different

selected

databases

to

return

different

numbers

of

documents

For

simplification

the

result

list

lengths

are

required

to

be

multiples

of

a

baseline

number

-LRB-

This

value

can

also

be

varied

but

for

simplification

it

is

set

to

in

this

paper

-RRB-

This

restriction

is

set

to

simulate

the

behavior

of

commercial

search

engines

on

the

Web

-LRB-

Search

engines

such

as

Google

and

AltaVista

return

only

or

document

ids

for

every

result

page

-RRB-

This

procedure

saves

the

computation

time

of

calculating

optimal

database

selection

by

allowing

the

step

of

dynamic

programming

to

be

instead

of

-LRB-

more

detail

is

discussed

latterly

-RRB-

For

further

simplification

we

restrict

to

select

at

most

documents

from

each

database

-LRB-

d

i

-RRB-

Then

the

selection

optimization

problem

is

formalized

as

follows

-RSB-

-LSB-

-RRB-

-LRB-

-RRB-

-LRB-

R

-RRB-

-LRB-

max

arg

k

k

d

N

d

N

d

I

to

Subject

d

d

I

d

i

rdoc

Total

i

i

sdb

i

i

i

d

j

ij

i

d

i

-LRB-

-RRB-

N

is

the

total

number

of

documents

to

be

retrieved

Unfortunately

there

is

no

simple

solution

for

this

optimization

problem

as

there

are

for

Equations

and

However

a

dynamic

programming

algorithm

can

be

applied

to

calculate

the

optimal

solution

The

basic

steps

of

this

dynamic

programming

method

are

described

in

Figure

As

this

algorithm

allows

retrieving

result

lists

of

varying

lengths

from

each

selected

database

it

is

called

UUM/HP-VL

algorithm

After

the

selection

decisions

are

made

the

selected

databases

are

searched

and

the

corresponding

document

ids

are

retrieved

from

each

database

The

final

step

of

document

retrieval

is

to

merge

the

returned

results

into

a

single

ranked

list

with

the

semi-supervised

learning

algorithm

It

was

pointed

out

before

that

the

SSL

algorithm

maps

the

database-specific

scores

into

the

centralized

document

scores

and

builds

the

final

ranked

list

accordingly

which

is

consistent

with

all

our

selection

procedures

where

documents

with

higher

probabilities

of

relevance

-LRB-

thus

higher

centralized

document

scores

-RRB-

are

selected

It

is

desirable

to

evaluate

distributed

information

retrieval

algorithms

with

testbeds

that

closely

simulate

the

real

world

applications

The

TREC

Web

collections

or

-LSB-

-RSB-

provide

a

way

to

partition

documents

by

different

Web

servers

In

this

way

a

large

number

-LRB-

O

-LRB-

-RRB-

-RRB-

of

databases

with

rather

diverse

contents

could

be

created

which

may

make

this

testbed

a

good

candidate

to

simulate

the

operational

environments

such

as

open

domain

hidden

Web

However

two

weakness

of

this

testbed

are

i

-RRB-

Each

database

contains

only

a

small

amount

of

document

-LRB-

documents

by

average

for

-RRB-

-LSB-

-RSB-

and

ii

-RRB-

The

contents

of

or

are

arbitrarily

crawled

from

the

Web

It

is

not

likely

for

a

hidden

Web

database

to

provide

personal

homepages

or

web

pages

indicating

that

the

pages

are

under

construction

and

there

is

no

useful

information

at

all

These

types

of

web

pages

are

contained

in

the

datasets

Therefore

the

noisy

Web

data

is

not

similar

with

that

of

high-quality

hidden

Web

database

contents

which

are

usually

organized

by

domain

experts

Another

choice

is

the

TREC

news/government

data

-LSB-

-RSB-

TREC

news/government

data

is

concentrated

on

relatively

narrow

topics

Compared

with

TREC

Web

data

i

-RRB-

The

news/government

documents

are

much

more

similar

to

the

contents

provided

by

a

topic-oriented

database

than

an

arbitrary

web

page

ii

-RRB-

A

database

in

this

testbed

is

larger

than

that

of

TREC

Web

data

By

average

a

database

contains

thousands

of

documents

which

is

more

realistic

than

a

database

of

TREC

Web

data

with

about

documents

As

the

contents

and

sizes

of

the

databases

in

the

TREC

news/government

testbed

are

more

similar

with

that

of

a

topic-oriented

database

it

is

a

good

candidate

to

simulate

the

distributed

information

retrieval

environments

of

large

organizations

-LRB-

companies

-RRB-

or

domain-specific

hidden

Web

sites

such

as

West

that

provides

access

to

legal

financial

and

news

text

databases

-LSB-

-RSB-

As

most

current

distributed

information

retrieval

systems

are

developed

for

the

environments

of

large

organizations

-LRB-

companies

-RRB-

or

domain-specific

hidden

Web

other

than

open

domain

hidden

Web

TREC

news/government

testbed

was

chosen

in

this

work

testbed

is

one

of

the

most

used

TREC

news/government

testbed

-LSB-

-RSB-

It

was

chosen

in

this

work

Three

testbeds

in

-LSB-

-RSB-

with

skewed

database

size

distributions

and

different

types

of

relevant

document

distributions

were

also

used

to

give

more

thorough

simulation

for

real

environments

databases

were

created

from

TREC

CDs

and

They

were

organized

by

source

and

publication

date

-LSB-

-RSB-

The

sizes

of

the

databases

are

not

skewed

Details

are

in

Table

Three

testbeds

built

in

-LSB-

-RSB-

were

based

on

the

testbed

Each

testbed

contains

many

small

databases

and

two

large

databases

created

by

merging

about

small

databases

together

Input

Complete

lists

of

probabilities

of

relevance

for

all

the

DB

databases

Output

Optimal

selection

solution

for

Equation

i

-RRB-

Create

the

three-dimensional

array

Sel

-LRB-

DB

N

N

sdb

-RRB-

Each

Sel

-LRB-

x

y

z

-RRB-

is

associated

with

a

selection

decision

xyz

d

which

represents

the

best

selection

decision

in

the

condition

only

databases

from

number

to

number

x

are

considered

for

selection

totally

y

documents

will

be

retrieved

only

z

databases

are

selected

out

of

the

x

database

candidates

And

Sel

-LRB-

x

y

z

-RRB-

is

the

corresponding

utility

value

by

choosing

the

best

selection

ii

-RRB-

Initialize

Sel

-LRB-

N

N

sdb

-RRB-

with

only

the

estimated

relevance

information

of

the

st

database

iii

-RRB-

Iterate

the

current

database

candidate

i

from

to

DB

For

each

entry

Sel

-LRB-

i

y

z

-RRB-

Find

k

such

that

-RRB-

min

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

max

arg

y

k

to

subject

d

R

z

k

y

i

Sel

k

k

j

ij

k

-RRB-

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

z

y

i

Sel

d

R

z

k

y

i

Sel

If

k

j

ij

gt

This

means

that

we

should

retrieve

k

documents

from

the

i

th

database

otherwise

we

should

not

select

this

database

and

the

previous

best

solution

Sel

-LRB-

y

z

-RRB-

should

be

kept

Then

set

the

value

of

iyz

d

and

Sel

-LRB-

i

y

z

-RRB-

accordingly

iv

-RRB-

The

best

selection

solution

is

given

by

Toral

rdoc

sdb

DB

N

N

d

and

the

corresponding

utility

value

is

Sel

-LRB-

DB

N

N

sdb

-RRB-

Figure

The

dynamic

programming

optimization

procedure

for

Equation

Testbed

statistics

Number

of

documents

Size

-LRB-

MB

-RRB-

Testbed

Size

-LRB-

GB

-RRB-

Min

Avg

Max

Min

Avg

Max

Query

set

statistics

Name

TREC

Topic

Set

TREC

Topic

Field

Average

Length

-LRB-

Words

-RRB-

Title

-LRB-

representative

-RRB-

The

databases

in

the

were

sorted

with

alphabetical

order

Two

large

databases

were

created

by

merging

small

databases

with

the

round-robin

method

Thus

the

two

large

databases

have

more

relevant

documents

due

to

their

large

sizes

even

though

the

densities

of

relevant

documents

are

roughly

the

same

as

the

small

databases

-LRB-

relevant

-RRB-

The

Associated

Press

collections

and

the

Wall

Street

Journal

collections

in

the

testbed

were

collapsed

into

two

large

databases

APall

and

WSJall

The

other

collections

were

left

unchanged

The

APall

and

WSJall

databases

have

higher

densities

of

documents

relevant

to

TREC

queries

than

the

small

databases

Thus

the

two

large

databases

have

many

more

relevant

documents

than

the

small

databases

-LRB-

nonrelevant

-RRB-

The

Federal

Register

collections

and

the

Department

of

Energy

collections

in

the

testbed

were

collapsed

into

two

large

databases

FRall

and

DOEall

The

other

collections

were

left

unchanged

The

FRall

and

DOEall

databases

have

lower

densities

of

documents

relevant

to

TREC

queries

than

the

small

databases

even

though

they

are

much

larger

queries

were

created

from

the

title

fields

of

TREC

topics

The

queries

were

used

as

training

queries

and

the

queries

were

used

as

test

queries

-LRB-

details

in

Table

-RRB-

Search

Engines

In

the

uncooperative

distributed

information

retrieval

environments

of

large

organizations

-LRB-

companies

-RRB-

or

domain-specific

hidden

Web

different

databases

may

use

different

types

of

search

engine

To

simulate

the

multiple

type-engine

environment

three

different

types

of

search

engines

were

used

in

the

experiments

INQUERY

-LSB-

-RSB-

a

unigram

statistical

language

model

with

linear

smoothing

-LSB-

-RSB-

and

a

TFIDF

retrieval

algorithm

with

ltc

weight

-LSB-

-RSB-

All

these

algorithms

were

implemented

with

the

Lemur

toolkit

-LSB-

-RSB-

These

three

kinds

of

search

engines

were

assigned

to

the

databases

among

the

four

testbeds

in

a

round-robin

manner

All

four

testbeds

described

in

Section

were

used

in

the

experiments

to

evaluate

the

resource

selection

effectiveness

of

the

database

recommendation

system

The

resource

descriptions

were

created

using

query-based

sampling

About

queries

were

sent

to

each

database

to

download

unique

documents

The

database

size

statistics

were

estimated

by

the

sample-resample

method

-LSB-

-RSB-

Fifty

queries

-LRB-

-RRB-

were

used

as

training

queries

to

build

the

relevant

logistic

model

and

to

fit

the

exponential

functions

of

the

centralized

document

score

curves

for

large

ratio

databases

-LRB-

details

in

Section

-RRB-

Another

queries

-LRB-

-RRB-

were

used

as

test

data

Resource

selection

algorithms

of

database

recommendation

systems

are

typically

compared

using

the

recall

metric

n

R

-LSB-

-RSB-

Let

B

denote

a

baseline

ranking

which

is

often

the

RBR

-LRB-

relevance

based

ranking

-RRB-

and

E

as

a

ranking

provided

by

a

resource

selection

algorithm

And

let

B

i

and

E

i

denote

the

number

of

relevant

documents

in

the

i

th

ranked

database

of

B

or

E

Then

R

n

is

defined

as

follows

k

i

i

k

i

i

k

B

E

R

-LRB-

-RRB-

Usually

the

goal

is

to

search

only

a

few

databases

so

our

figures

only

show

results

for

selecting

up

to

databases

The

experiments

summarized

in

Figure

compared

the

effectiveness

of

the

three

resource

selection

algorithms

namely

the

CORI

ReDDE

and

UUM/HR

The

UUM/HR

algorithm

is

described

in

Section

It

can

be

seen

from

Figure

that

the

ReDDE

and

UUM/HR

algorithms

are

more

effective

-LRB-

on

the

representative

relevant

and

nonrelevant

testbeds

-RRB-

or

as

good

as

-LRB-

on

the

testbed

-RRB-

the

CORI

resource

selection

algorithm

The

UUM/HR

algorithm

is

more

effective

than

the

ReDDE

algorithm

on

the

representative

and

relevant

testbeds

and

is

about

the

same

as

the

ReDDE

algorithm

on

the

and

the

nonrelevant

testbeds

This

suggests

that

the

UUM/HR

algorithm

is

more

robust

than

the

ReDDE

algorithm

It

can

be

noted

that

when

selecting

only

a

few

databases

on

the

or

the

nonrelevant

testbeds

the

ReDEE

algorithm

has

a

small

advantage

over

the

UUM/HR

algorithm

We

attribute

this

to

two

causes

i

-RRB-

The

ReDDE

algorithm

was

tuned

on

the

testbed

and

ii

-RRB-

Although

the

difference

is

small

this

may

suggest

that

our

logistic

model

of

estimating

probabilities

of

relevance

is

not

accurate

enough

More

training

data

or

a

more

sophisticated

model

may

help

to

solve

this

minor

puzzle

Collections

Selected

Collections

Selected

Testbed

Representative

Testbed

Collection

Selected

Collection

Selected

Relevant

Testbed

Nonrelevant

Testbed

Figure

Resource

selection

experiments

on

the

four

testbeds

For

document

retrieval

the

selected

databases

are

searched

and

the

returned

results

are

merged

into

a

single

final

list

In

all

of

the

experiments

discussed

in

this

section

the

results

retrieved

from

individual

databases

were

combined

by

the

semi-supervised

learning

results

merging

algorithm

This

version

of

the

SSL

algorithm

-LSB-

-RSB-

is

allowed

to

download

a

small

number

of

returned

document

texts

on

the

fly

to

create

additional

training

data

in

the

process

of

learning

the

linear

models

which

map

database-specific

document

scores

into

estimated

centralized

document

scores

It

has

been

shown

to

be

very

effective

in

environments

where

only

short

result-lists

are

retrieved

from

each

selected

database

-LSB-

-RSB-

This

is

a

common

scenario

in

operational

environments

and

was

the

case

for

our

experiments

Document

retrieval

effectiveness

was

measured

by

Precision

at

the

top

part

of

the

final

document

list

The

experiments

in

this

section

were

conducted

to

study

the

document

retrieval

effectiveness

of

five

selection

algorithms

namely

the

CORI

ReDDE

UUM/HR

UUM/HP-FL

and

UUM/HP-VL

algorithms

The

last

three

algorithms

were

proposed

in

Section

All

the

first

four

algorithms

selected

or

databases

and

documents

were

retrieved

from

each

selected

database

The

UUM/HP-FL

algorithm

also

selected

or

databases

but

it

was

allowed

to

adjust

the

number

of

documents

to

retrieve

from

each

selected

database

the

number

retrieved

was

constrained

to

be

from

to

and

a

multiple

of

The

and

representative

testbeds

were

selected

for

document

retrieval

as

they

represent

two

extreme

cases

of

resource

selection

effectiveness

in

one

case

the

CORI

algorithm

is

as

good

as

the

other

algorithms

and

in

the

other

case

it

is

quite

Table

Precision

on

the

representative

testbed

when

databases

were

selected

-LRB-

The

first

baseline

is

CORI

the

second

baseline

for

UUM/HP

methods

is

UUM/HR

-RRB-

Precision

at

Doc

Rank

CORI

ReDDE

UUM/HR

UUM/HP-FL

UUM/HP-VL

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

Precision

on

the

representative

testbed

when

databases

were

selected

-LRB-

The

first

baseline

is

CORI

the

second

baseline

for

UUM/HP

methods

is

UUM/HR

-RRB-

Precision

at

Doc

Rank

CORI

ReDDE

UUM/HR

UUM/HP-FL

UUM/HP-VL

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

Precision

on

the

testbed

when

databases

were

selected

-LRB-

The

first

baseline

is

CORI

the

second

baseline

for

UUM/HP

methods

is

UUM/HR

-RRB-

Precision

at

Doc

Rank

CORI

ReDDE

UUM/HR

UUM/HP-FL

UUM/HP-VL

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

Precision

on

the

testbed

when

databases

were

selected

-LRB-

The

first

baseline

is

CORI

the

second

baseline

for

UUM/HP

methods

is

UUM/HR

-RRB-

Precision

at

Doc

Rank

CORI

ReDDE

UUM/HR

UUM/HP-FL

UUM/HP-VL

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

a

lot

worse

than

the

other

algorithms

Tables

and

show

the

results

on

the

testbed

and

Tables

and

show

the

results

on

the

representative

testbed

On

the

testbed

the

document

retrieval

effectiveness

of

the

CORI

selection

algorithm

is

roughly

the

same

or

a

little

bit

better

than

the

ReDDE

algorithm

but

both

of

them

are

worse

than

the

other

three

algorithms

-LRB-

Tables

and

-RRB-

The

UUM/HR

algorithm

has

a

small

advantage

over

the

CORI

and

ReDDE

algorithms

One

main

difference

between

the

UUM/HR

algorithm

and

the

ReDDE

algorithm

was

pointed

out

before

The

UUM/HR

uses

training

data

and

linear

interpolation

to

estimate

the

centralized

document

score

curves

while

the

ReDDE

algorithm

-LSB-

-RSB-

uses

a

heuristic

method

assumes

the

centralized

document

score

curves

are

step

functions

and

makes

no

distinction

among

the

top

part

of

the

curves

This

difference

makes

UUM/HR

better

than

the

ReDDE

algorithm

at

distinguishing

documents

with

high

probabilities

of

relevance

from

low

probabilities

of

relevance

Therefore

the

UUM/HR

reflects

the

high-precision

retrieval

goal

better

than

the

ReDDE

algorithm

and

thus

is

more

effective

for

document

retrieval

The

UUM/HR

algorithm

does

not

explicitly

optimize

the

selection

decision

with

respect

to

the

high-precision

goal

as

the

UUM/HP-FL

and

UUM/HP-VL

algorithms

are

designed

to

do

It

can

be

seen

that

on

this

testbed

the

UUM/HP-FL

and

UUM/HP-VL

algorithms

are

much

more

effective

than

all

the

other

algorithms

This

indicates

that

their

power

comes

from

explicitly

optimizing

the

high-precision

goal

of

document

retrieval

in

Equations

and

On

the

representative

testbed

CORI

is

much

less

effective

than

other

algorithms

for

distributed

document

retrieval

-LRB-

Tables

and

-RRB-

The

document

retrieval

results

of

the

ReDDE

algorithm

are

better

than

that

of

the

CORI

algorithm

but

still

worse

than

the

results

of

the

UUM/HR

algorithm

On

this

testbed

the

three

UUM

algorithms

are

about

equally

effective

Detailed

analysis

shows

that

the

overlap

of

the

selected

databases

between

the

UUM/HR

UUM/HP-FL

and

UUM/HP-VL

algorithms

is

much

larger

than

the

experiments

on

the

testbed

since

all

of

them

tend

to

select

the

two

large

databases

This

explains

why

they

are

about

equally

effective

for

document

retrieval

In

real

operational

environments

databases

may

return

no

document

scores

and

report

only

ranked

lists

of

results

As

the

unified

utility

maximization

model

only

utilizes

retrieval

scores

of

sampled

documents

with

a

centralized

retrieval

algorithm

to

calculate

the

probabilities

of

relevance

it

makes

database

selection

decisions

without

referring

to

the

document

scores

from

individual

databases

and

can

be

easily

generalized

to

this

case

of

rank

lists

without

document

scores

The

only

adjustment

is

that

the

SSL

algorithm

merges

ranked

lists

without

document

scores

by

assigning

the

documents

with

pseudo-document

scores

normalized

for

their

ranks

-LRB-

In

a

ranked

list

of

documents

the

first

one

has

a

score

of

the

second

has

a

score

of

etc

-RRB-

which

has

been

studied

in

-LSB-

-RSB-

The

experiment

results

on

testbed

with

selected

databases

are

shown

in

Table

The

experiment

setting

was

the

same

as

before

except

that

the

document

scores

were

eliminated

intentionally

and

the

selected

databases

only

return

ranked

lists

of

document

ids

It

can

be

seen

from

the

results

that

the

UUM/HP-FL

and

UUM/HP-VL

work

well

with

databases

returning

no

document

scores

and

are

still

more

effective

than

other

alternatives

Other

experiments

with

databases

that

return

no

document

scores

are

not

reported

but

they

show

similar

results

to

prove

the

effectiveness

of

UUM/HP-FL

and

UUM/HP-VL

algorithms

The

above

experiments

suggest

that

it

is

very

important

to

optimize

the

high-precision

goal

explicitly

in

document

retrieval

The

new

algorithms

based

on

this

principle

achieve

better

or

at

least

as

good

results

as

the

prior

state-of-the-art

algorithms

in

several

environments

Distributed

information

retrieval

solves

the

problem

of

finding

information

that

is

scattered

among

many

text

databases

on

local

area

networks

and

Internets

Most

previous

research

use

effective

resource

selection

algorithm

of

database

recommendation

system

for

distributed

document

retrieval

application

We

argue

that

the

high-recall

resource

selection

goal

of

database

recommendation

and

high-precision

goal

of

document

retrieval

are

related

but

not

identical

This

kind

of

inconsistency

has

also

been

observed

in

previous

work

but

the

prior

solutions

either

used

heuristic

methods

or

assumed

cooperation

by

individual

databases

-LRB-

e.g.

all

the

databases

used

the

same

kind

of

search

engines

-RRB-

which

is

frequently

not

true

in

the

uncooperative

environment

In

this

work

we

propose

a

unified

utility

maximization

model

to

integrate

the

resource

selection

of

database

recommendation

and

document

retrieval

tasks

into

a

single

unified

framework

In

this

framework

the

selection

decisions

are

obtained

by

optimizing

different

objective

functions

As

far

as

we

know

this

is

the

first

work

that

tries

to

view

and

theoretically

model

the

distributed

information

retrieval

task

in

an

integrated

manner

The

new

framework

continues

a

recent

research

trend

studying

the

use

of

query-based

sampling

and

a

centralized

sample

database

A

single

logistic

model

was

trained

on

the

centralized

Table

Precision

on

the

testbed

when

databases

were

selected

-LRB-

The

first

baseline

is

CORI

the

second

baseline

for

UUM/HP

methods

is

UUM/HR

-RRB-

-LRB-

Search

engines

do

not

return

document

scores

-RRB-

Precision

at

Doc

Rank

CORI

ReDDE

UUM/HR

UUM/HP-FL

UUM/HP-VL

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

docs

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

sample

database

to

estimate

the

probabilities

of

relevance

of

documents

by

their

centralized

retrieval

scores

while

the

centralized

sample

database

serves

as

a

bridge

to

connect

the

individual

databases

with

the

centralized

logistic

model

Therefore

the

probabilities

of

relevance

for

all

the

documents

across

the

databases

can

be

estimated

with

very

small

amount

of

human

relevance

judgment

which

is

much

more

efficient

than

previous

methods

that

build

a

separate

model

for

each

database

This

framework

is

not

only

more

theoretically

solid

but

also

very

effective

One

algorithm

for

resource

selection

-LRB-

UUM/HR

-RRB-

and

two

algorithms

for

document

retrieval

-LRB-

UUM/HP-FL

and

UUM/HP-VL

-RRB-

are

derived

from

this

framework

Empirical

studies

have

been

conducted

on

testbeds

to

simulate

the

distributed

search

solutions

of

large

organizations

-LRB-

companies

-RRB-

or

domain-specific

hidden

Web

Furthermore

the

UUM/HP-FL

and

UUM/HP-VL

resource

selection

algorithms

are

extended

with

a

variant

of

SSL

results

merging

algorithm

to

address

the

distributed

document

retrieval

task

when

selected

databases

do

not

return

document

scores

Experiments

have

shown

that

these

algorithms

achieve

results

that

are

at

least

as

good

as

the

prior

state-of-the-art

and

sometimes

considerably

better

Detailed

analysis

indicates

that

the

advantage

of

these

algorithms

comes

from

explicitly

optimizing

the

goals

of

the

specific

tasks

The

unified

utility

maximization

framework

is

open

for

different

extensions

When

cost

is

associated

with

searching

the

online

databases

the

utility

framework

can

be

adjusted

to

automatically

estimate

the

best

number

of

databases

to

search

so

that

a

large

amount

of

relevant

documents

can

be

retrieved

with

relatively

small

costs

Another

extension

of

the

framework

is

to

consider

the

retrieval

effectiveness

of

the

online

databases

which

is

an

important

issue

in

the

operational

environments

All

of

these

are

the

directions

of

future

research

This

research

was

supported

by

NSF

grants

and

Any

opinions

findings

conclusions

or

recommendations

expressed

in

this

paper

are

the

authors

and

do

not

necessarily

reflect

those

of

the

sponsor

-LSB-

-RSB-

J.

Callan

-LRB-

-RRB-

Distributed

information

retrieval

In

W.B.

Croft

editor

Advances

in

Information

Retrieval

Kluwer

Academic

Publishers

-LRB-

pp.

-RRB-

-LSB-

-RSB-

J.

Callan

W.B.

Croft

and

J.

Broglio

-LRB-

-RRB-

TREC

and

TIPSTER

experiments

with

INQUERY

Information

Processing

and

Management

-LRB-

-RRB-

-LRB-

pp.

-RRB-

-LSB-

-RSB-

J.

G.

Conrad

X.

S.

Guo

P.

Jackson

and

M.

Meziou

-LRB-

-RRB-

Database

selection

using

actual

physical

and

acquired

logical

collection

resources

in

a

massive

domain-specific

operational

environment

Distributed

search

over

the

hidden

web

Hierarchical

database

sampling

and

selection

In

Proceedings

of

the

th

International

Conference

on

Very

Large

Databases

-LRB-

VLDB

-RRB-

-LSB-

-RSB-

N.

Craswell

-LRB-

-RRB-

Methods

for

distributed

information

retrieval

Ph.

D.

thesis

The

Australian

Nation

University

-LSB-

-RSB-

N.

Craswell

D.

Hawking

and

P.

Thistlewaite

-LRB-

-RRB-

Merging

results

from

isolated

search

engines

In

Proceedings

of

Australasian

Database

Conference

-LSB-

-RSB-

D.

D'Souza

J.

Thom

and

J.

Zobel

-LRB-

-RRB-

A

comparison

of

techniques

for

selecting

text

collections

In

Proceedings

of

the

Australasian

Database

Conference

-LSB-

-RSB-

N.

Fuhr

-LRB-

-RRB-

A

Decision-Theoretic

approach

to

database

selection

in

networked

IR

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LRB-

pp.

-RRB-

-LSB-

-RSB-

L.

Gravano

C.

Chang

H.

Garcia-Molina

and

A.

Paepcke

-LRB-

-RRB-

STARTS

Stanford

proposal

for

internet

meta-searching

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

-LSB-

-RSB-

L.

Gravano

P.

Ipeirotis

and

M.

Sahami

-LRB-

-RRB-

QProber

A

System

for

Automatic

Classification

of

Hidden-Web

Databases

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LSB-

-RSB-

P.

Ipeirotis

and

L.

Gravano

-LRB-

-RRB-

Distributed

search

over

the

hidden

web

Hierarchical

database

sampling

and

selection

In

Proceedings

of

the

International

Conference

on

Very

Large

Databases

-LRB-

VLDB

-RRB-

-LSB-

-RSB-

InvisibleWeb.com

-LSB-

-RSB-

The

lemur

toolkit

-LSB-

-RSB-

J.

Lu

and

J.

Callan

-LRB-

-RRB-

Content-based

information

retrieval

in

peer-to-peer

networks

In

Proceedings

of

the

International

Conference

on

Information

and

Knowledge

Management

-LSB-

-RSB-

W.

Meng

C.T.

Yu

and

K.L.

Liu

-LRB-

-RRB-

Building

efficient

and

effective

metasearch

engines

ACM

Comput

Surv

-LRB-

-RRB-

-LSB-

-RSB-

H.

Nottelmann

and

N.

Fuhr

-LRB-

-RRB-

Evaluating

different

method

of

estimating

retrieval

quality

for

resource

selection

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

H.

Nottelmann

and

N.

Fuhr

-LRB-

-RRB-

The

MIND

architecture

for

heterogeneous

multimedia

federated

digital

libraries

ACM

SIGIR

Workshop

on

Distributed

Information

Retrieval

-LSB-

-RSB-

A.L.

Powell

J.C.

French

J.

Callan

M.

Connell

and

C.L.

Viles

-LRB-

-RRB-

The

impact

of

database

selection

on

distributed

searching

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

A.L.

Powell

and

J.C.

French

-LRB-

-RRB-

Comparing

the

performance

of

database

selection

algorithms

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LRB-

pp.

-RRB-

-LSB-

-RSB-

C.

Sherman

-LRB-

-RRB-

Search

for

the

invisible

web

Guardian

Unlimited

-LSB-

-RSB-

L.

Si

and

J.

Callan

-LRB-

-RRB-

Using

sampled

data

and

regression

to

merge

search

engine

results

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

L.

Si

and

J.

Callan

-LRB-

-RRB-

Relevant

document

distribution

estimation

method

for

resource

selection

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

L.

Si

and

J.

Callan

-LRB-

-RRB-

A

Semi-Supervised

learning

method

to

merge

search

engine

results

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LRB-

pp.

-RRB-

resource selection
distributed information retrieval