Using

Web

Helper

Agent

Profiles

in

Query

Generation

Personalized

information

agents

can

help

overcome

some

of

the

limitations

of

communal

Web

information

sources

such

as

portals

and

search

engines

Two

important

components

of

these

agents

are

user

profiles

and

information

filtering

or

gathering

services

Ideally

these

components

can

be

sep-arated

so

that

a

single

user

profile

can

be

leveraged

for

a

variety

of

information

services

Toward

that

end

we

are

building

an

information

agent

called

SurfAgent

in

previous

studies

we

have

developed

and

tested

methods

for

automatically

learning

a

user

profile

-LSB-

-RSB-

In

this

paper

we

evaluate

alternative

methods

for

recommending

new

documents

to

a

user

by

generating

queries

from

the

user

profile

and

submitting

them

to

a

popular

search

engine

Our

study

focuses

on

three

questions

How

do

different

algorithms

for

query

generation

perform

relative

to

each

other

Is

positive

relevance

feedback

adequate

to

support

the

task

Can

a

user

profile

be

learned

independent

of

the

service

We

found

that

three

algorithms

appear

to

excel

and

that

using

only

positive

feedback

does

degrade

the

results

somewhat

We

conclude

with

the

results

of

a

pilot

user

study

for

assessing

interaction

of

the

profile

and

the

query

generation

mechanisms

H.

-LSB-

Information

storage

and

retrieval

-RSB-

Information

Search

and

Retrieval

Query

formulation

Search

process

Information

filtering

The

Web

has

become

an

indispensable

source

of

information

for

many

people

Based

on

surveys

of

the

most

popular

Web

sites

-LSB-

-RSB-

users

deal

with

the

overwhelming

amount

and

constantly

updating

nature

of

the

information

by

routinely

visiting

hub

sites

-LRB-

e.g.

Netscape

Yahoo

CNN

-RRB-

and

making

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

AAMAS

July

Melbourne

Australia

Copyright

ACM

copious

use

of

search

engines

-LRB-

e.g.

AltaVista

Excite

Magellan

-RRB-

Users

have

derived

tremendous

leverage

from

shared

information

resources

such

as

those

just

mentioned

Hubs

or

portals

provide

communally

useful

information

about

perennial

-LRB-

e.g.

financial

management

child

rearing

-RRB-

and

timely

-LRB-

e.g.

September

events

stock

quote

-RRB-

topics

Search

engines

satisfy

specific

spontaneous

information

needs

As

our

appetite

for

information

increases

so

does

its

availability

on

the

Web

Studies

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

have

identified

limitations

with

these

tools

for

satisfying

users

needs

for

example

users

appear

to

lack

the

motivation

to

learn

how

to

formulate

complex

queries

or

to

view

long

lists

of

potential

matches

Meta-search

engines

such

as

Meta-Crawler

-LSB-

-RSB-

SavvySearch

-LSB-

-RSB-

and

NECI

-LSB-

-RSB-

propose

to

overcome

the

Web

coverage

problem

by

combining

the

indexing

power

of

multiple

stand-alone

search

engines

However

because

they

leverage

the

capabilities

of

many

search

engines

they

tend

to

generalize

the

search

task

limiting

the

access

to

search-engine

specific

advanced

search

capabilities

and

perhaps

introducing

even

more

noise

into

the

return

results

One

promising

approach

to

compensating

for

the

limitations

is

to

personalize

the

tools

Pretschner

and

Gauch

divide

personalization

into

two

types

personalized

access

to

resources

and

filtering/ranking

-LSB-

-RSB-

For

example

My

Yahoo

-LRB-

-RRB-

provides

personalized

access

by

allowing

users

to

design

their

own

Yahoo

page

with

pertinent

information

many

search

and

meta-search

engines

support

some

customization

-LRB-

e.g.

types

of

search

return

amount

and

search

engine

selection

-RRB-

Softbot

s

or

information

agents

augment

searching

and

information

gathering

-LRB-

filtering/ranking

-RRB-

Personalized

information

agents

such

as

Letizia

-LSB-

-RSB-

WebWatcher

-LSB-

-RSB-

and

WebMate

-LSB-

-RSB-

can

provide

a

range

of

services

from

automatically

retrieving

Web

pages

to

assisting

in

formulating

queries

These

agents

generally

comply

with

the

architecture

presented

in

Figure

The

agent

intercedes

between

the

user

and

their

Web

access

monitoring

the

user

's

activities

to

construct

a

model

of

user

requests

-LRB-

the

profile

-RRB-

to

be

used

for

specific

information

services

-LRB-

e.g.

modifying

requests

and/or

filtering

results

-RRB-

In

our

research

we

adopt

the

principle

that

user

overhead

needs

to

be

minimized

The

profile

should

be

learned

by

asking

the

user

to

simply

click

on

a

feedback

button

positioned

on

the

bottom

of

each

page

to

indicate

interest

Learning

should

track

changes

in

user

interests

The

profile

should

support

multiple

information

services

monitor

generate

request

for

doc

stream

filter

user

Web

user

profile

document

flow

command

flow

Figure

Architecture

of

a

Web

information

helper

agent

In

previous

papers

we

have

assessed

some

alternative

approaches

to

learning

user

profiles

-LSB-

-RSB-

In

this

paper

we

examine

alternative

approaches

to

one

of

the

services

query

generation

to

find

new

documents

-LRB-

i.e.

automatically

retrieving

Web

pages

that

match

a

user

's

interests

by

submitting

queries

to

a

Web

search

engine

-RRB-

In

particular

we

are

interested

in

answering

the

following

questions

How

do

different

algorithms

for

query

generation

perform

relative

to

each

other

For

our

case

query

generation

involves

constructing

queries

from

a

user

profile

that

are

submitted

to

a

search

engine

for

the

purpose

of

harvesting

documents

Is

positive

relevance

feedback

adequate

to

support

the

task

To

minimize

user

overhead

we

have

solicited

only

positive

relevance

feedback

Obviously

this

provides

relatively

weak

knowledge

requiring

the

profiling

mechanism

to

self-organize

the

categories

of

interest

and

to

trade-off

precision

Can

a

user

profile

be

learned

independent

of

the

service

If

so

then

user

overhead

can

be

minimized

and

multiple

services

provided

as

part

of

the

same

agent

This

paper

describes

a

framework

for

evaluating

alternative

algorithms

for

information

gathering

agents

and

a

study

that

was

designed

to

address

the

three

questions

above

In

summary

we

found

Three

algorithms

perform

best

in

terms

of

extracting

sufficient

numbers

of

documents

from

the

search

engine

and

in

terms

of

the

relevance

of

the

links

returned

We

did

find

evidence

that

soliciting

only

positive

feedback

hampers

query

generation

however

it

is

not

clear

that

the

degradation

in

performance

is

worth

the

cost

of

obtaining

the

negative

feedback

As

often

happens

the

study

raised

some

issues

that

are

still

to

be

resolved

-LRB-

particularly

about

the

evaluation

criteria

and

the

interaction

of

profiling

and

query

generation

-RRB-

we

conclude

with

a

pilot

study

in

which

we

investigate

how

to

resolve

these

issues

SurfAgent

-LSB-

-RSB-

is

a

personalized

Web

information

agent

which

follows

the

basic

architecture

in

Figure

It

is

designed

as

a

testbed

for

expediting

plug-and-play

and

evaluation

of

alternative

algorithms

front-ends

and

service

tasks

Its

two

primary

components

are

the

user

profile

and

the

module

which

generates

requests

for

document

streams

Monitoring

should

be

simple

and

unobtrusive

Filtering

depends

on

the

representation

and

construction

of

the

user

profile

forcing

a

relatively

tight

coupling

of

those

two

components

This

section

provides

an

overview

of

its

user

profiling

and

document

stream

generation

Building

User

Profiles

and

Filtering

The

user

profile

maintained

by

a

Web

helper

agent

is

a

model

of

what

documents

the

user

finds

relevant

Like

most

other

personal

Web

helper

agents

SurfAgent

uses

TF-IDF

vectors

-LSB-

-RSB-

as

the

basis

of

its

user

profile

representation

One

such

vector

is

used

to

represent

each

of

the

several

different

topics

of

interest

associated

with

each

user

Over

time

topic

descriptions

are

learned

from

user-supplied

examples

of

relevant

documents

which

are

added

to

the

existing

TF-IDF

vectors

in

what

is

known

as

relevance

feedback

-LSB-

-RSB-

Associated

with

each

vector

is

a

dissemination

threshold

which

is

used

when

new

documents

are

filtered

by

the

agent

if

the

similarity

between

the

new

document

's

vector

and

a

vector

in

the

profile

exceeds

the

associated

dissemination

threshold

the

document

is

considered

relevant

and

shown

to

the

user

We

found

that

learning

the

appropriate

dissemination

threshold

was

critical

to

filtering

performance

and

that

one

could

be

learned

with

relatively

little

feedback

-LRB-

i.e.

relevance

judgments

-RRB-

-LSB-

-RSB-

TF-IDF

vectors

and

their

associated

dissemination

thresholds

are

known

in

the

Information

Retrieval

-LRB-

IR

-RRB-

literature

as

filtering

queries

This

type

of

query

is

distinguished

from

a

typical

retrieval

query

-LRB-

used

with

search

engines

or

at

a

library

-RRB-

by

a

few

characteristics

Filtering

queries

tend

to

be

used

repeatedly

over

a

long

period

of

time

during

which

they

can

be

improved

and

maintained

through

learning

and

relevance

feedback

whereas

retrieval

queries

are

typically

used

only

once

Also

filtering

queries

typically

contain

lots

of

terms

with

complex

weighting

schemes

whereas

retrieval

queries

tend

to

be

a

boolean

combination

of

relatively

few

terms

with

no

weighting

at

all

Each

filtering

query

in

SurfAgent

's

user

profile

corresponds

to

a

distinct

topic

of

interest

User

profiles

are

learned

in

one

of

two

ways

First

relevant

documents

provided

as

training

by

the

user

can

be

explicitly

associated

with

the

topic

of

interest

Alternatively

to

minimize

overhead

to

the

user

incremental

clustering

can

be

used

by

the

agent

to

automatically

classify

relevant

examples

into

internally

learned

topics

-LSB-

-RSB-

In

the

latter

situation

the

user

only

needs

to

prompt

the

agent

when

a

relevant

document

has

been

encountered

without

having

to

associate

it

with

a

particular

topic

of

interest

To

minimize

user

disruption

we

request

only

positive

examples

We

augmented

existing

IR

clustering

techniques

to

accommodate

Web

needs

-LRB-

i.e.

avoid

storing

the

documents

themselves

require

minimal

user

overhead

and

be

associated

with

a

user

-RRB-

In

our

earlier

study

we

found

that

a

tuned

version

of

the

Doubling

algorithm

-LSB-

-RSB-

achieved

high

recall

without

a

great

sacrifice

on

precision

Incoming

Document

Streams

Personal

information

agents

use

a

wide

range

of

techniques

to

generate

incoming

streams

Letizia

pre-fetches

Web

pages

by

exploring

the

links

in

the

Web

page

cur-rently

being

viewed

Similarly

WebWatcher

analyzes

text

in

and

around

links

in

the

current

page

to

predict

relevant

links

worth

following

Fab

builds

a

list

of

likely

to

be

relevant

documents

through

best-first

search

documents

that

pass

the

filtering

phase

are

then

included

in

a

list

of

User

User

Profile

Extract

Query

Search

Engine

Filter

Documents

Figure

Incoming

document

streams

generated

by

querying

a

search

engine

mended

links

Finally

WebMate

filters

articles

found

on

a

list

of

well-known

news

sources

in

order

to

compile

a

personal

newspaper

for

its

user

Our

goals

are

to

maximize

the

quality

of

the

incoming

document

stream

generated

for

SurfAgent

while

at

the

same

time

minimizing

effort

For

this

purpose

a

promising

technique

appears

to

be

the

construction

of

queries

that

are

suitable

for

a

large-scale

search

engine

such

as

Google

-LSB-

-RSB-

Well-formulated

queries

have

the

potential

to

off-load

significant

portions

of

the

filtering

task

to

the

search

engine

which

should

provide

the

agent

with

a

document

stream

consisting

of

more

relevant

documents

In

this

paper

we

explore

several

methods

of

generating

search

engine

queries

from

the

user

profile

of

a

personal

Web

helper

agent

We

wish

to

find

both

the

method

and

the

query

length

that

would

optimize

the

relevance

of

the

documents

returned

by

the

search

engine

with

respect

to

the

user

profile

This

process

is

illustrated

in

Figure

Filtering

queries

are

not

directly

suitable

for

being

submitted

to

a

search

engine

They

are

complex

models

representing

a

possibly

large

collection

of

documents

they

contain

a

large

number

of

terms

with

associated

weights

which

would

overly

restrict

the

range

of

documents

a

search

engine

might

return

Query

generation

techniques

have

evolved

from

the

more

general

query

refinement

mechanism

of

relevance

feedback

-LSB-

-RSB-

For

instance

in

-LSB-

-RSB-

search

engine

queries

are

extended

with

features

extracted

from

positive

and

negative

examples

in

order

to

bias

them

toward

a

more

relevant

sub-topic

Several

other

researchers

have

been

concerned

with

extracting

only

a

few

highly

representative

terms

from

the

representation

of

a

large

document

cluster

For

WebACE

-LSB-

-RSB-

the

authors

propose

a

mechanism

for

generating

queries

from

a

cluster

of

documents

based

on

the

average

word

count

-LRB-

term

frequency

TF

-RRB-

of

the

terms

contained

in

the

documents

of

the

cluster

and

the

document

frequency

-LRB-

DF

-RRB-

i.e.

the

number

of

documents

in

the

cluster

that

contain

a

specified

term

A

set

of

k

terms

with

the

highest

TF

and

another

set

of

k

terms

with

the

highest

DF

are

selected

from

the

cluster

The

intersection

of

these

two

sets

was

submitted

to

Yahoo

search

as

a

query

yielding

a

total

of

results

which

were

later

refined

to

by

augmenting

the

query

with

terms

resulting

from

the

difference

between

the

TF

and

DF

sets

CorpusBuilder

-LSB-

-RSB-

uses

automatic

query

generation

to

collect

documents

in

a

specified

language

-LRB-

Slovenian

-RRB-

from

the

Web

with

the

purpose

of

building

a

corpus

in

that

language

The

point

is

to

preserve

computation

power

by

avoiding

a

brute-force

crawl

of

the

Web

where

an

expensive

classifier

would

have

to

be

run

on

each

encountered

document

By

generating

queries

from

the

already

existing

corpus

the

authors

hope

to

significantly

increase

the

likelihood

that

the

resulting

documents

would

already

be

in

Slovenian

thus

speeding

up

document

collection

Several

methods

for

generating

queries

are

used

interchangeably

uniform

select

n

terms

from

the

relevant

documents

with

equal

probability

term-frequency

select

n

most

frequent

terms

from

the

relevant

documents

probabilistic

term-frequency

select

n

terms

from

the

relevant

documents

with

probability

proportional

to

their

term

frequency

odds-ratio

select

n

terms

with

highest

odds-ratio

scores

as

given

by

the

formula

OR

t

log

P

-LRB-

t

relevant

-RRB-

-LRB-

P

-LRB-

t

nonrelevant

-RRB-

-RRB-

P

-LRB-

t

nonrelevant

-RRB-

-LRB-

P

-LRB-

t

relevant

-RRB-

-RRB-

where

t

is

the

term

and

P

-LRB-

t

relevant

-RRB-

and

P

-LRB-

t

nonrelevant

-RRB-

are

the

probabilities

of

t

appearing

in

a

relevant

and

non-relevant

document

respectively

probabilistic

odds-ratio

select

n

terms

with

probability

proportional

to

their

odds-ratio

score

The

authors

report

best

results

with

n

and

the

simple

odds-ratio

method

However

this

method

is

not

necessarily

applicable

to

our

task

because

identifying

relevance

with

respect

to

a

query

cluster

is

somewhat

more

subtle

than

determining

whether

a

returned

document

is

in

a

particular

language

such

as

Slovenian

The

purpose

of

this

study

is

to

examine

the

role

of

query

generation

technique

for

a

Web

information

agent

what

techniques

work

well

how

much

user

overhead

is

warranted

and

how

query

generation

interacts

with

profiling

These

three

factors

correspond

to

the

three

questions

articulated

in

the

Introduction

Experiment

Design

The

basic

protocol

for

our

study

was

as

follows

construct

user

profiles

generate

queries

from

those

profiles

using

each

of

the

query

generation

methods

submit

queries

of

different

lengths

to

Google

evaluate

the

results

This

led

to

a

factorial

experiment

in

which

the

independent

variables

were

query

generation

method

and

query

length

and

the

dependent

variables

were

two

evaluation

metrics

return

count

and

relevance

Constructing

the

profiles

To

expedite

experimental

control

and

data

collection

we

constructed

two

user

profiles

from

TREC

disk

TREC

data

consist

of

a

large

number

of

articles

-LRB-

over

-RRB-

of

which

a

large

portion

have

been

judged

as

either

relevant

or

non-relevant

with

respect

to

a

set

of

different

topics

Our

first

topic

is

on

airport

security

measures

and

was

constructed

based

on

relevant

documents

which

appeared

in

the

Los

Angeles

Times

over

the

course

of

two

years

-LRB-

-RRB-

this

topic

will

be

referred

to

as

LATIMES

The

second

topic

is

on

genetic

research

and

was

constructed

based

on

relevant

documents

from

the

Foreign

Broadcasting

Information

Service

appeared

in

this

topic

will

be

referred

to

as

FBIS

One

topic

was

picked

randomly

from

each

of

the

two

document

collection

on

the

TREC

disk

We

used

synthetically

generated

topics

in

order

to

test

the

hypothetical

scenario

in

which

negative

feedback

is

available

By

default

SurfAgent

does

not

collect

negative

feedback

in

the

form

of

documents

which

are

non-relevant

to

a

given

topic

Thus

we

are

interested

in

how

much

performance

might

be

sacrificed

by

restricting

feedback

to

only

positive

examples

The

number

of

positive

documents

used

in

the

construction

of

each

topic

-LRB-

and

respectively

-RRB-

is

realistic

compared

to

what

a

human

user

would

have

been

capable

of

providing

while

building

her

profile

in

real

life

Generating

Queries

We

implemented

several

methods

including

both

methods

which

use

such

negative

examples

-LRB-

e.g.

odds-ratio

-RRB-

against

methods

which

do

not

-LRB-

e.g.

Boley

's

method

-LSB-

-RSB-

and

term

frequency

based

methods

-RRB-

In

addition

to

the

methods

mentioned

in

Section

we

add

two

methods

deterministic

extraction

of

highest

weight

terms

for

SurfAgent

's

TF-IDF

profile

vectors

and

probabilistic

weighted

extraction

from

the

TF-IDF

profile

vectors

The

complete

list

of

methods

used

is

given

below

Uniform

-LRB-

Unif

-RRB-

baseline

case

select

n

terms

with

uniform

probability

Boley

select

the

intersection

of

the

k

top

ranking

terms

according

to

term

frequency

in

one

set

and

document

frequency

in

the

other

TF

select

n

top

ranking

terms

according

to

term

frequency

Probabilistic

TF

-LRB-

P-TF

-RRB-

select

n

terms

with

probability

proportional

to

their

term

frequency

OR

select

the

top

ranking

n

terms

according

to

their

odds-ratio

score

Probabilistic

OR

-LRB-

P-OR

-RRB-

select

n

terms

with

probability

proportional

to

their

odds-ratio

score

TFIDF

select

n

terms

with

the

highest

TF-IDF

weight

from

the

profile

vector

Probabilistic

TF-IDF

-LRB-

P-TFIDF

-RRB-

select

n

terms

with

probability

proportional

to

their

TF-IDF

weights

Text

REtrieval

Conference

TREC

benchmark

disks

are

publicly

available

and

can

be

ordered

from

the

conference

homepage

at

The

probabilistic

versions

were

included

because

injection

of

small

amounts

of

randomness

has

been

found

to

be

helpful

in

other

search

problems

Code

from

SurfAgent

was

modified

to

collect

the

data

required

by

all

query

generation

methods

employed

For

each

topic

we

collected

the

following

data

average

term

frequencies

for

terms

in

relevant

documents

document

frequencies

for

terms

in

relevant

documents

TF-IDF

vector

built

from

relevant

documents

odds-ratio

scores

for

terms

in

relevant

documents

-LRB-

odds-ratio

scores

are

based

on

both

relevant

and

non-relevant

documents

related

to

the

topic

-RRB-

From

these

data

we

generated

queries

of

seven

lengths

-LRB-

two

to

eight

terms

-RRB-

for

each

of

the

eight

methods

For

the

four

probabilistic

methods

we

generated

queries

of

each

length

which

means

their

reported

results

will

be

averages

over

those

queries

For

Boley

's

method

we

repeatedly

computed

the

intersection

of

the

sets

consisting

of

the

top

k

ranking

terms

w.r.t.

TF

and

DF

while

incrementing

k

We

retained

all

distinct

queries

of

length

between

and

For

FBIS

no

value

of

k

generated

a

query

of

length

Similarly

for

LATIMES

there

was

no

value

of

k

resulting

in

a

query

of

length

Submit

the

Queries

The

previous

step

resulted

in

queries

-LRB-

for

each

topic

-RRB-

We

submitted

these

queries

to

the

Google

search

engine

and

collected

back

the

first

page

of

results

By

default

a

page

can

contain

up

to

responses

Collect

the

results

The

results

of

the

queries

-LRB-

the

URLs

returned

-RRB-

were

parsed

out

of

the

page

returned

by

Google

and

their

corresponding

documents

were

retrieved

from

the

Web

and

stored

locally

We

discarded

-LRB-

and

decremented

our

hit

count

accordingly

-RRB-

all

dead

links

and

all

hits

that

were

in

a

format

other

than

ASCII

or

PDF

a

total

of

out

of

hits

were

discarded

The

PDF

documents

were

converted

into

ASCII

using

the

pdftotext

utility

Results

For

each

valid

hit

we

computed

the

similarity

between

the

document

's

TF-IDF

vector

and

the

TF-IDF

vector

of

the

appropriate

topic

which

is

a

measure

of

the

document

's

relevance

For

each

combination

of

query

generation

method

and

query

length

we

recorded

the

number

of

hits

received

the

relevance

of

the

best

hit

and

the

average

relevance

over

all

hits

received

For

the

probabilistic

methods

these

measurements

represent

average

values

obtained

over

the

ten

repetitions

for

each

combination

of

parameters

The

results

are

summarized

in

Table

for

the

FBIS

topic

and

Table

for

the

LATIMES

topic

The

three

rows

corresponding

to

each

method

indicate

average

relevance

-LRB-

top

-RRB-

maximum

relevance

and

number

of

returned

hits

-LRB-

bottom

-RRB-

All

methods

return

at

least

seven

documents

with

query

lengths

of

but

most

taper

off

in

the

number

returned

ASCII

includes

all

variants

and

versions

of

HTML

XML

etc.

query

length

method

avg

Unif

max

cnt

avg

P-TF

max

cnt

avg

P

OR

max

cnt

avg

P-TFIDF

max

cnt

avg

Boley

max

cnt

avg

TF

max

cnt

avg

OR

max

cnt

avg

TFIDF

max

cnt

Table

Average

relevance

Maximum

relevance

and

count

of

returned

hits

for

the

FBIS

topic

on

genetic

technology

query

length

method

avg

Unif

max

cnt

avg

P-TF

max

cnt

avg

P

OR

max

cnt

avg

P-TFIDF

max

cnt

avg

Boley

max

cnt

avg

TF

max

cnt

avg

OR

max

cnt

avg

TFIDF

max

cnt

Table

Average

relevance

Maximum

relevance

and

count

of

returned

hits

for

the

LATIMES

topic

on

airport

security

with

longer

query

lengths

For

the

deterministic

methods

the

relevance

increases

as

the

query

length

increases

-LRB-

until

or

-RRB-

but

the

relevance

for

the

probabilistic

methods

tends

to

plateau

early

All

methods

consistently

outperform

the

baseline

uniform

term

selection

Probabilistic

methods

are

outperformed

by

Figure

Box

plot

of

relevance

by

method

for

FBIS

topic

at

query

length

Figure

Box

plot

of

relevance

by

method

for

FBIS

topic

at

query

length

the

non-probabilistic

ones

which

is

consistent

with

the

observations

in

-LSB-

-RSB-

The

best

results

for

the

FBIS

topic

were

obtained

using

TFIDF

at

query

length

a

total

of

hits

were

returned

with

an

average

relevance

of

and

a

maximum

relevance

of

The

best

results

for

the

LATIMES

topic

were

obtained

using

OR

at

query

length

two

hits

were

returned

with

average

relevance

of

and

maximum

relevance

of

Query

lengths

and

were

the

only

ones

where

all

methods

lead

to

non-empty

returns

for

both

topics

To

test

whether

the

differences

between

the

methods

were

significant

we

ran

an

analysis

of

variance

-LRB-

ANOVA

-RRB-

study

on

each

topic

at

query

lengths

and

with

the

query

generation

method

as

the

independent

variable

and

relevance

as

the

dependent

The

effects

were

significant

in

each

case

for

FBIS

we

obtained

F

p

at

query

length

and

F

p

at

query

length

for

LATIMES

we

obtained

F

p

at

query

length

and

F

p

at

query

length

Box

plots

of

relevance

by

method

for

query

lengths

and

are

given

in

Figures

and

for

FBIS

and

Figures

and

Figure

Box

plot

of

relevance

by

method

for

LATIMES

topic

at

query

length

Figure

Box

plot

of

relevance

by

method

for

LATIMES

topic

at

query

length

for

LATIMES

Note

that

medians

rather

than

means

are

marked

on

the

graph

These

plots

illustrate

several

points

obscured

by

the

previous

table

First

while

TFIDF

returns

the

best

match

and

second

best

median

in

all

cases

overall

better

results

are

obtained

using

OR

for

FBIS

and

TF

and

Boley

for

LATIMES

Second

each

method

can

show

high

variance

in

the

results

although

TFIDF

tends

generally

to

higher

variance

Additionally

the

results

for

query

length

have

higher

variance

than

those

for

query

length

Finally

the

distributions

are

often

skewed

For

example

Boley

consistently

has

a

higher

median

than

mean

Because

relevance

for

all

returned

documents

is

measured

against

the

TF-IDF

vector

of

the

corresponding

topic

the

experiments

are

slightly

biased

in

favor

of

the

TFIDF

query

generation

method

Our

experiments

can

not

prove

that

TFIDF

query

generation

is

sufficient

but

its

good

performance

coupled

with

the

not

always

good

performance

of

OR

suggest

that

we

do

not

incur

a

significant

loss

by

leaving

out

negative

feedback

Collecting

other

information

based

on

positive

feedback

in

addition

to

TF-IDF

topic

vectors

may

be

required

with

SurfAgent

e.g.

straight

TF

vectors

and

query

length

method

TFIDF

nrel

cnt

P-TFIDF

nrel

cnt

Table

Number

of

relevant

documents

and

count

of

returned

hits

for

the

user-generated

topic

on

stress

relief

topic-specific

document

frequency

information

would

allow

us

to

use

TF

and

Boley

query

generation

in

addition

to

the

TFIDF

method

As

the

results

show

sometimes

TF

and

Boley

perform

better

than

OR

and

TFIDF

Both

Boley

and

TFIDF

consistently

result

in

many

links

being

returned

even

for

long

query

lengths

Many

hits

are

desirable

when

the

agent

will

be

pruning

out

previously

visited

sites

and

filtering

the

returned

links

according

to

a

filtering

threshold

The

computation

time

required

for

query

generation

by

each

of

the

studied

methods

is

negligible

when

compared

to

the

network-related

delays

incurred

while

downloading

the

actual

documents

To

gain

an

understanding

of

the

performance

of

TFIDF

query

generation

without

the

bias

present

in

our

experiments

with

synthetically

generated

topics

we

have

also

performed

a

pilot

study

with

a

user-generated

topic

containing

documents

on

stress

relief

Since

SurfAgent

only

collects

TF-IDF

information

at

this

time

query

generation

was

limited

to

the

TFIDF

and

P-TFIDF

methods

We

followed

the

same

protocol

as

with

the

synthetically

generated

topics

query

lengths

were

between

and

and

results

were

aver-aged

over

ten

trials

for

the

probabilistic

version

P-TFIDF

A

total

of

distinct

URLs

were

returned

by

Google

We

shuffled

these

URLs

and

asked

the

user

to

examine

each

one

of

them

and

mark

the

ones

she

found

to

be

relevant

to

her

topic

documents

out

of

the

total

of

were

found

relevant

Table

presents

the

number

of

relevant

documents

and

the

number

of

hits

returned

for

each

parameter

combination

This

pilot

study

supports

the

hypothesis

that

TFIDF

based

queries

will

generate

an

adequate

incoming

stream

queries

of

length

up

to

six

returned

at

least

nine

hits

from

Google

Unlike

the

previous

study

the

shorter

queries

yielded

lower

relevance

which

could

be

due

to

the

way

the

user

was

judging

relevance

or

to

the

nature

of

the

topic

As

a

followup

we

will

be

designing

a

user

study

that

includes

the

three

apparently

best

methods

-LRB-

TF

TFIDF

and

Boley

-RRB-

We

will

focus

on

three

issues

Does

method

performance

vary

among

users

and

topics

-LRB-

as

is

suggested

by

our

current

study

-RRB-

Should

profile

construction

incorporate

more

information

Does

relevance

assessment

change

as

profiles

become

more

mature

Can

the

best

query

length

be

determined

a

priori

We

studied

several

methods

for

generating

queries

from

a

user

profile

to

answer

three

questions

related

to

the

design

of

Web

information

agents

First

how

do

different

query

generation

algorithms

perform

relative

to

each

other

In

fact

we

observed

significantly

different

performance

among

the

eight

methods

tested

Overall

Boley

TFIDF

and

to

a

lesser

extent

TF

provided

a

good

number

of

hits

and

relatively

high

relevance

Second

is

positive

relevance

feedback

adequate

to

support

the

task

We

found

that

leaving

out

negative

training

examples

does

not

incur

a

significant

performance

loss

Odds-Ratio

was

found

to

excel

on

one

topic

but

its

competitive

advantage

does

not

appear

to

be

worth

the

additional

overhead

expected

from

the

user

TFIDF

and

Boley

requiring

only

positive

relevance

feedback

generated

queries

that

resulted

in

relevant

hits

Third

can

user

profiles

be

learned

independently

of

the

service

The

results

from

TFIDF

and

the

pilot

experiment

do

suggest

it

However

the

pilot

study

also

suggests

that

either

user

relevance

judgments

may

be

a

bit

lower

-LRB-

harsher

-RRB-

than

the

automated

method

or

that

the

profile

may

not

adequately

reflect

the

users

interests

In

fact

the

good

performance

of

Boley

and

TF

indicates

that

in

some

cases

it

might

be

worthwhile

to

collect

more

than

TF-IDF

information

from

the

user-supplied

positive

training

examples

This

last

question

will

be

examined

in

more

detail

in

the

future

Our

study

confirmed

that

additional

user

burden

in

the

form

of

negative

feedback

appears

unwarranted

to

support

document

generation

and

that

queries

generated

based

on

automatically

learned

profiles

can

guide

harvesting

of

new

documents

of

interest

This

last

result

is

excellent

news

for

the

development

of

agents

that

leverage

a

single

learned

profile

to

personalize

a

multitude

of

web

information

services

This

research

was

supported

in

part

by

National

Science

Foundation

Career

Award

The

United

States

Government

is

authorized

to

reproduce

and

distribute

reprints

for

governmental

purposes

notwithstanding

any

copyright

notation

herein

-LSB-

-RSB-

R.

Armstrong

D.

Freitag

T.

Joachims

and

T.

Mitchell

WebWatcher

A

Learning

Apprentice

for

the

World

Wide

Web

In

Proceedings

of

the

AAAI

Spring

Symposium

on

Information

Gathering

from

Heterogeneous

Distributed

Resources

Stanford

CA

-LSB-

-RSB-

D.

Boley

M.

Gini

R.

Gross

E.

Han

K.

Hastingsand

G.

Karypis

V.

Kumar

M.

Bamshad

and

J.

Moore

Document

Categorization

and

Query

Generation

on

the

World

Wide

Web

Using

WebAce

AI

Review

-LRB-

-RRB-

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

Anatomy

of

a

Large-scale

Hypertextual

Web

Search

Engine

Computer

Networks

and

ISDN

Systems

pages

-LSB-

-RSB-

M.

Charikar

C.

Chekuri

T.

Feder

and

R.

Motwani

Incremental

Clustering

and

Dynamic

Information

Retrieval

Proceedings

of

the

ACM

Symposium

on

Theory

of

Computing

-LSB-

-RSB-

L.

Chen

and

Katia

Sycara

WebMate

A

Personal

Agent

for

Browsing

and

Searching

In

Proceedings

of

the

Second

International

Conference

on

Autonomous

Agents

Minneapolis

MN

-LSB-

-RSB-

D.

Dreilinger

and

A.E.

Howe

Experiences

with

selecting

search

engines

using

meta-search

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LSB-

-RSB-

G.W.

Flake

E.J.

Glover

S.

Lawrence

and

C.L.

Giles

Extracting

Query

Modifications

from

Nonlinear

SVMs

In

Proceedings

of

the

Eleventh

International

World

Wide

Web

Conference

-LRB-

WWW

-RRB-

Honolulu

HI

U.S.A.

-LSB-

-RSB-

R.

Ghani

R.

Jones

and

D.

Mladenic

On-line

learning

for

query

generation

Finding

documents

matching

a

minority

concept

on

the

web

In

Proc

of

the

First

Asia-Pacific

Conference

on

Web

Intelligence

-LSB-

-RSB-

E.J.

Glover

G.W.

Flake

S.

Lawrence

W.P.

Birmingham

A.

Kruger

C.L.

Giles

and

D.

Pennock

Improving

Category

Specific

Web

Search

by

Learning

Query

Modifications

In

Proceedings

of

the

IEEE

Symposium

on

Applications

and

the

Internet

-LRB-

SAINT

-RRB-

San

Diego

CA

U.S.A.

-LSB-

-RSB-

T.

Joachims

D.

Freitag

and

T.

Mitchell

WebWatcher

A

Tour

Guide

for

the

World

Wide

Web

In

Proc

of

the

International

Joint

Conference

on

Artificial

Intelligence

Nagoya

Japan

-LSB-

-RSB-

S.

Lawrence

and

C.L.

Giles

Context

and

page

analysis

for

improved

web

search

IEEE

Internet

Computing

-LRB-

-RRB-

-LSB-

-RSB-

S.

Lawrence

and

C.L.

Giles

Searching

the

world

wide

web

Science

April

-LSB-

-RSB-

H.

Lieberman

Letizia

An

agent

that

assists

web

browsing

In

Proceedings

of

the

International

Joint

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

Montreal

Canada

-LSB-

-RSB-

Inc.

Netmation

most

popular

web

sites

-LSB-

-RSB-

A.

Pretschner

and

S.

Gauch

Personalization

on

the

web

Technical

Report

Dept.

of

Electrical

Engineering

and

Computer

Science

University

of

Kansas

December

-LSB-

-RSB-

J.J.

Rocchio

Relevance

feedback

in

information

retrieval

In

G.

Salton

editor

The

SMART

Retrieval

System

Experiments

in

Automatic

Document

Processing

Prentice-Hall

-LSB-

-RSB-

G.

Salton

Automatic

Text

Processing

The

Transformation

Analysis

and

Retrieval

of

Information

by

Computer

Addison-Wesley

-LSB-

-RSB-

E.

Selberg

and

O.

Etzioni

The

metacrawler

architecture

for

resource

aggregation

on

the

web

IEEE

Expert

-LRB-

-RRB-

-LSB-

-RSB-

G.L.

Somlo

and

A.E.

Howe

Adaptive

lightweight

text

filtering

In

Proceedings

of

the

Conference

on

Intelligent

Data

Analysis

-LRB-

IDA

-RRB-

Lisbon

Portugal

September

-LSB-

-RSB-

Gabriel

L.

Somlo

and

Adele

E.

Howe

Incremental

clustering

for

profile

maintenance

in

information

gathering

web

agents

In

Proceedings

of

the

International

Conference

on

Autonomous

Agents

-LRB-

AGENTS

-RRB-

Montreal

Canada

May

-LSB-

-RSB-

A.

Spink

J.

Bateman

and

B.J.

Jansen

Searching

heterogeneous

collections

on

the

web

Behavior

of

excite

users

Information

Research

An

Electronic

Journal

-LRB-

-RRB-

O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O