Implicit

User

Modeling

for

Personalized

Search

Information

retrieval

systems

-LRB-

e.g.

web

search

engines

-RRB-

are

critical

for

overcoming

information

overload

A

major

deficiency

of

existing

retrieval

systems

is

that

they

generally

lack

user

modeling

and

are

not

adaptive

to

individual

users

resulting

in

inherently

non-optimal

retrieval

performance

For

example

a

tourist

and

a

programmer

may

use

the

same

word

java

to

search

for

different

information

but

the

current

search

systems

would

return

the

same

results

In

this

paper

we

study

how

to

infer

a

user

's

interest

from

the

user

's

search

context

and

use

the

inferred

implicit

user

model

for

personalized

search

We

present

a

decision

theoretic

framework

and

develop

techniques

for

implicit

user

modeling

in

information

retrieval

We

develop

an

intelligent

client-side

web

search

agent

-LRB-

UCAIR

-RRB-

that

can

perform

eager

implicit

feedback

e.g.

query

expansion

based

on

previous

queries

and

immediate

result

reranking

based

on

clickthrough

information

Experiments

on

web

search

show

that

our

search

agent

can

improve

search

accuracy

over

the

popular

Google

search

engine

H.

-LSB-

Information

Search

and

Retrieval

-RSB-

Retrieval

models

Relevance

feedback

Search

Process

Algorithms

Although

many

information

retrieval

systems

-LRB-

e.g.

web

search

engines

and

digital

library

systems

-RRB-

have

been

successfully

deployed

the

current

retrieval

systems

are

far

from

optimal

A

major

deficiency

of

existing

retrieval

systems

is

that

they

generally

lack

user

modeling

and

are

not

adaptive

to

individual

users

-LSB-

-RSB-

This

inherent

non-optimality

is

seen

clearly

in

the

following

two

cases

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

CIKM

October

November

Bremen

Germany

Copyright

ACM

-LRB-

-RRB-

Different

users

may

use

exactly

the

same

query

-LRB-

e.g.

Java

-RRB-

to

search

for

different

information

-LRB-

e.g.

the

Java

island

in

Indonesia

or

the

Java

programming

language

-RRB-

but

existing

IR

systems

return

the

same

results

for

these

users

Without

considering

the

actual

user

it

is

impossible

to

know

which

sense

Java

refers

to

in

a

query

-LRB-

-RRB-

A

user

's

information

needs

may

change

over

time

The

same

user

may

use

Java

sometimes

to

mean

the

Java

island

in

Indonesia

and

some

other

times

to

mean

the

programming

language

Without

recognizing

the

search

context

it

would

be

again

impossible

to

recognize

the

correct

sense

In

order

to

optimize

retrieval

accuracy

we

clearly

need

to

model

the

user

appropriately

and

personalize

search

according

to

each

individual

user

The

major

goal

of

user

modeling

for

information

retrieval

is

to

accurately

model

a

user

's

information

need

which

is

unfortunately

a

very

difficult

task

Indeed

it

is

even

hard

for

a

user

to

precisely

describe

what

his/her

information

need

is

What

information

is

available

for

a

system

to

infer

a

user

's

information

need

Obviously

the

user

's

query

provides

the

most

direct

evidence

Indeed

most

existing

retrieval

systems

rely

solely

on

the

query

to

model

a

user

's

information

need

However

since

a

query

is

often

extremely

short

the

user

model

constructed

based

on

a

keyword

query

is

inevitably

impoverished

An

effective

way

to

improve

user

modeling

in

information

retrieval

is

to

ask

the

user

to

explicitly

specify

which

documents

are

relevant

-LRB-

i.e.

useful

for

satisfying

his/her

information

need

-RRB-

and

then

to

improve

user

modeling

based

on

such

examples

of

relevant

documents

This

is

called

relevance

feedback

which

has

been

proved

to

be

quite

effective

for

improving

retrieval

accuracy

-LSB-

-RSB-

Unfortunately

in

real

world

applications

users

are

usually

reluctant

to

make

the

extra

effort

to

provide

relevant

examples

for

feedback

-LSB-

-RSB-

It

is

thus

very

interesting

to

study

how

to

infer

a

user

's

information

need

based

on

any

implicit

feedback

information

which

naturally

exists

through

user

interactions

and

thus

does

not

require

any

extra

user

effort

Indeed

several

previous

studies

have

shown

that

implicit

user

modeling

can

improve

retrieval

accuracy

In

-LSB-

-RSB-

a

web

browser

-LRB-

Curious

Browser

-RRB-

is

developed

to

record

a

user

's

explicit

relevance

ratings

of

web

pages

-LRB-

relevance

feedback

-RRB-

and

browsing

behavior

when

viewing

a

page

such

as

dwelling

time

mouse

click

mouse

movement

and

scrolling

-LRB-

implicit

feedback

-RRB-

It

is

shown

that

the

dwelling

time

on

a

page

amount

of

scrolling

on

a

page

and

the

combination

of

time

and

scrolling

have

a

strong

correlation

with

explicit

relevance

ratings

which

suggests

that

implicit

feedback

may

be

helpful

for

inferring

user

information

need

In

-LSB-

-RSB-

user

clickthrough

data

is

collected

as

training

data

to

learn

a

retrieval

function

which

is

used

to

produce

a

customized

ranking

of

search

results

that

suits

a

group

of

users

preferences

In

-LSB-

-RSB-

the

clickthrough

data

collected

over

a

long

time

period

is

exploited

through

query

expansion

to

improve

retrieval

accuracy

While

a

user

may

have

general

long

term

interests

and

preferences

for

information

often

he/she

is

searching

for

documents

to

satisfy

an

ad

hoc

information

need

which

only

lasts

for

a

short

period

of

time

once

the

information

need

is

satisfied

the

user

would

generally

no

longer

be

interested

in

such

information

For

example

a

user

may

be

looking

for

information

about

used

cars

in

order

to

buy

one

but

once

the

user

has

bought

a

car

he/she

is

generally

no

longer

interested

in

such

information

In

such

cases

implicit

feedback

information

collected

over

a

long

period

of

time

is

unlikely

to

be

very

useful

but

the

immediate

search

context

and

feedback

information

such

as

which

of

the

search

results

for

the

current

information

need

are

viewed

can

be

expected

to

be

much

more

useful

Consider

the

query

Java

again

Any

of

the

following

immediate

feedback

information

about

the

user

could

potentially

help

determine

the

intended

meaning

of

Java

in

the

query

-LRB-

-RRB-

The

previous

query

submitted

by

the

user

is

hashtable

-LRB-

as

opposed

to

e.g.

travel

Indonesia

-RRB-

-LRB-

-RRB-

In

the

search

results

the

user

viewed

a

page

where

words

such

as

programming

software

and

applet

occur

many

times

To

the

best

of

our

knowledge

how

to

exploit

such

immediate

and

short-term

search

context

to

improve

search

has

so

far

not

been

well

addressed

in

the

previous

work

In

this

paper

we

study

how

to

construct

and

update

a

user

model

based

on

the

immediate

search

context

and

implicit

feedback

information

and

use

the

model

to

improve

the

accuracy

of

ad

hoc

retrieval

In

order

to

maximally

benefit

the

user

of

a

retrieval

system

through

implicit

user

modeling

we

propose

to

perform

eager

implicit

feedback

That

is

as

soon

as

we

observe

any

new

piece

of

evidence

from

the

user

we

would

update

the

system

's

belief

about

the

user

's

information

need

and

respond

with

improved

retrieval

results

based

on

the

updated

user

model

We

present

a

decision-theoretic

framework

for

optimizing

interactive

information

retrieval

based

on

eager

user

model

updating

in

which

the

system

responds

to

every

action

of

the

user

by

choosing

a

system

action

to

optimize

a

utility

function

In

a

traditional

retrieval

paradigm

the

retrieval

problem

is

to

match

a

query

with

documents

and

rank

documents

according

to

their

relevance

values

As

a

result

the

retrieval

process

is

a

simple

independent

cycle

of

query

and

result

display

In

the

proposed

new

retrieval

paradigm

the

user

's

search

context

plays

an

important

role

and

the

inferred

implicit

user

model

is

exploited

immediately

to

benefit

the

user

The

new

retrieval

paradigm

is

thus

fundamentally

different

from

the

traditional

paradigm

and

is

inherently

more

general

We

further

propose

specific

techniques

to

capture

and

exploit

two

types

of

implicit

feedback

information

-LRB-

-RRB-

identifying

related

immediately

preceding

query

and

using

the

query

and

the

corresponding

search

results

to

select

appropriate

terms

to

expand

the

current

query

and

-LRB-

-RRB-

exploiting

the

viewed

document

summaries

to

immediately

rerank

any

documents

that

have

not

yet

been

seen

by

the

user

Using

these

techniques

we

develop

a

client-side

web

search

agent

UCAIR

-LRB-

User-Centered

Adaptive

Information

Retrieval

-RRB-

on

top

of

a

popular

search

engine

-LRB-

Google

-RRB-

Experiments

on

web

search

show

that

our

search

agent

can

improve

search

accuracy

over

Google

Since

the

implicit

information

we

exploit

already

naturally

exists

through

user

interactions

the

user

does

not

need

to

make

any

extra

effort

Thus

the

developed

search

agent

can

improve

existing

web

search

performance

without

additional

effort

from

the

user

The

remaining

sections

are

organized

as

follows

In

Section

we

discuss

the

related

work

In

Section

we

present

a

decision-theoretic

interactive

retrieval

framework

for

implicit

user

modeling

In

Section

we

present

the

design

and

implementation

of

an

intelligent

client-side

web

search

agent

-LRB-

UCAIR

-RRB-

that

performs

eager

implicit

feedback

In

Section

we

report

our

experiment

results

using

the

search

agent

Section

concludes

our

work

Implicit

user

modeling

for

personalized

search

has

been

studied

in

previous

work

but

our

work

differs

from

all

previous

work

in

several

aspects

-LRB-

-RRB-

We

emphasize

the

exploitation

of

immediate

search

context

such

as

the

related

immediately

preceding

query

and

the

viewed

documents

in

the

same

session

while

most

previous

work

relies

on

long-term

collection

of

implicit

feedback

information

-LSB-

-RSB-

-LRB-

-RRB-

We

perform

eager

feedback

and

bring

the

benefit

of

implicit

user

modeling

as

soon

as

any

new

implicit

feedback

information

is

available

while

the

previous

work

mostly

exploits

long-term

implicit

feedback

-LSB-

-RSB-

-LRB-

-RRB-

We

propose

a

retrieval

framework

to

integrate

implicit

user

modeling

with

the

interactive

retrieval

process

while

the

previous

work

either

studies

implicit

user

modeling

separately

from

retrieval

-LSB-

-RSB-

or

only

studies

specific

retrieval

models

for

exploiting

implicit

feedback

to

better

match

a

query

with

documents

-LSB-

-RSB-

-LRB-

-RRB-

We

develop

and

evaluate

a

personalized

Web

search

agent

with

online

user

studies

while

most

existing

work

evaluates

algorithms

offline

without

real

user

interactions

Currently

some

search

engines

provide

rudimentary

personalization

such

as

Google

Personalized

web

search

-LSB-

-RSB-

which

allows

users

to

explicitly

describe

their

interests

by

selecting

from

predefined

topics

so

that

those

results

that

match

their

interests

are

brought

to

the

top

and

My

Yahoo!

search

-LSB-

-RSB-

which

gives

users

the

option

to

save

web

sites

they

like

and

block

those

they

dislike

In

contrast

UCAIR

personalizes

web

search

through

implicit

user

modeling

without

any

additional

user

efforts

Furthermore

the

personalization

of

UCAIR

is

provided

on

the

client

side

There

are

two

remarkable

advantages

on

this

First

the

user

does

not

need

to

worry

about

the

privacy

infringement

which

is

a

big

concern

for

personalized

search

-LSB-

-RSB-

Second

both

the

computation

of

personalization

and

the

storage

of

the

user

profile

are

done

at

the

client

side

so

that

the

server

load

is

reduced

dramatically

-LSB-

-RSB-

There

have

been

many

works

studying

user

query

logs

-LSB-

-RSB-

or

query

dynamics

-LSB-

-RSB-

UCAIR

makes

direct

use

of

a

user

's

query

history

to

benefit

the

same

user

immediately

in

the

same

search

session

UCAIR

first

judges

whether

two

neighboring

queries

belong

to

the

same

information

session

and

if

so

it

selects

terms

from

the

previous

query

to

perform

query

expansion

Our

query

expansion

approach

is

similar

to

automatic

query

expansion

-LSB-

-RSB-

but

instead

of

using

pseudo

feedback

to

expand

the

query

we

use

user

's

implicit

feedback

information

to

expand

the

current

query

These

two

techniques

may

be

combined

In

interactive

IR

a

user

interacts

with

the

retrieval

system

through

an

action

dialogue

in

which

the

system

responds

to

each

user

action

with

some

system

action

For

example

the

user

's

action

may

be

submitting

a

query

and

the

system

's

response

may

be

returning

a

list

of

document

summaries

In

general

the

space

of

user

actions

and

system

responses

and

their

granularities

would

depend

on

the

interface

of

a

particular

retrieval

system

In

principle

every

action

of

the

user

can

potentially

provide

new

evidence

to

help

the

system

better

infer

the

user

's

information

need

Thus

in

order

to

respond

optimally

the

system

should

use

all

the

evidence

collected

so

far

about

the

user

when

choosing

a

response

When

viewed

in

this

way

most

existing

search

engines

are

clearly

non-optimal

For

example

if

a

user

has

viewed

some

documents

on

the

first

page

of

search

results

when

the

user

clicks

on

the

Next

link

to

fetch

more

results

an

existing

retrieval

system

would

still

return

the

next

page

of

results

retrieved

based

on

the

original

query

without

considering

the

new

evidence

that

a

particular

result

has

been

viewed

by

the

user

We

propose

to

optimize

retrieval

performance

by

adapting

system

responses

based

on

every

action

that

a

user

has

taken

and

cast

the

optimization

problem

as

a

decision

task

Specifically

at

any

time

the

system

would

attempt

to

do

two

tasks

-LRB-

-RRB-

User

model

updating

Monitor

any

useful

evidence

from

the

user

regarding

his/her

information

need

and

update

the

user

model

as

soon

as

such

evidence

is

available

-LRB-

-RRB-

Improving

search

results

Rerank

immediately

all

the

documents

that

the

user

has

not

yet

seen

as

soon

as

the

user

model

is

updated

We

emphasize

eager

updating

and

reranking

which

makes

our

work

quite

different

from

any

existing

work

Below

we

present

a

formal

decision

theoretic

framework

for

optimizing

retrieval

performance

through

implicit

user

modeling

in

interactive

information

retrieval

A

decision-theoretic

framework

Let

A

be

the

set

of

all

user

actions

and

R

-LRB-

a

-RRB-

be

the

set

of

all

possible

system

responses

to

a

user

action

a

A

At

any

time

let

A

t

-LRB-

a

a

t

-RRB-

be

the

observed

sequence

of

user

actions

so

far

-LRB-

up

to

time

point

t

-RRB-

and

R

-LRB-

r

r

-RRB-

be

the

responses

that

the

system

has

made

responding

to

the

user

actions

The

system

's

goal

is

to

choose

an

optimal

response

r

t

R

-LRB-

a

t

-RRB-

for

the

current

user

action

a

t

Let

M

be

the

space

of

all

possible

user

models

We

further

define

a

loss

function

L

-LRB-

a

r

m

-RRB-

where

a

A

is

a

user

action

r

R

-LRB-

a

-RRB-

is

a

system

response

and

m

M

is

a

user

model

L

-LRB-

a

r

m

-RRB-

encodes

our

decision

preferences

and

assesses

the

optimality

of

responding

with

r

when

the

current

user

model

is

m

and

the

current

user

action

is

a

According

to

Bayesian

decision

theory

the

optimal

decision

at

time

t

is

to

choose

a

response

that

minimizes

the

Bayes

risk

i.e.

r

t

argmin

rR

-LRB-

a

t

-RRB-

M

L

-LRB-

a

t

r

m

t

-RRB-

P

-LRB-

m

t

U

D

A

t

R

-RRB-

dm

t

-LRB-

-RRB-

where

P

-LRB-

m

t

U

D

A

t

R

-RRB-

is

the

posterior

probability

of

the

user

model

m

t

given

all

the

observations

about

the

user

U

we

have

made

up

to

time

t.

To

simplify

the

computation

of

Equation

let

us

assume

that

the

posterior

probability

mass

P

-LRB-

m

t

U

D

A

t

R

-RRB-

is

mostly

concentrated

on

the

mode

m

t

argmax

m

t

P

-LRB-

m

t

U

D

A

t

R

-RRB-

We

can

then

approximate

the

integral

with

the

value

of

the

loss

function

at

m

t

That

is

r

t

argmin

rR

-LRB-

a

t

-RRB-

L

-LRB-

a

t

r

m

t

-RRB-

-LRB-

-RRB-

where

m

t

argmax

m

t

P

-LRB-

m

t

U

D

A

t

R

-RRB-

Leaving

aside

how

to

define

and

estimate

these

probabilistic

models

and

the

loss

function

we

can

see

that

such

a

decision-theoretic

formulation

suggests

that

in

order

to

choose

the

optimal

response

to

a

t

the

system

should

perform

two

tasks

-LRB-

-RRB-

compute

the

current

user

model

and

obtain

m

t

based

on

all

the

useful

information

-LRB-

-RRB-

choose

a

response

r

t

to

minimize

the

loss

function

value

L

-LRB-

a

t

r

t

m

t

-RRB-

When

a

t

does

not

affect

our

belief

about

m

t

the

first

step

can

be

omitted

and

we

may

reuse

m

for

m

t

Note

that

our

framework

is

quite

general

since

we

can

potentially

model

any

kind

of

user

actions

and

system

responses

In

most

cases

as

we

may

expect

the

system

's

response

is

some

ranking

of

documents

i.e.

for

most

actions

a

R

-LRB-

a

-RRB-

consists

of

all

the

possible

rankings

of

the

unseen

documents

and

the

decision

problem

boils

down

to

choosing

the

best

ranking

of

unseen

documents

based

on

the

most

current

user

model

When

a

is

the

action

of

submitting

a

keyword

query

such

a

response

is

exactly

what

a

current

retrieval

system

would

do

However

we

can

easily

imagine

that

a

more

intelligent

web

search

engine

would

respond

to

a

user

's

clicking

of

the

Next

link

-LRB-

to

fetch

more

unseen

results

-RRB-

with

a

more

opti-mized

ranking

of

documents

based

on

any

viewed

documents

in

the

current

page

of

results

In

fact

according

to

our

eager

updating

strategy

we

may

even

allow

a

system

to

respond

to

a

user

's

clicking

of

browser

's

Back

button

after

viewing

a

document

in

the

same

way

so

that

the

user

can

maximally

benefit

from

implicit

feedback

These

are

precisely

what

our

UCAIR

system

does

User

models

A

user

model

m

M

represents

what

we

know

about

the

user

U

so

in

principle

it

can

contain

any

information

about

the

user

that

we

wish

to

model

We

now

discuss

two

important

components

in

a

user

model

The

first

component

is

a

component

model

of

the

user

's

information

need

Presumably

the

most

important

factor

affecting

the

optimality

of

the

system

's

response

is

how

well

the

response

addresses

the

user

's

information

need

Indeed

at

any

time

we

may

assume

that

the

system

has

some

belief

about

what

the

user

is

interested

in

which

we

model

through

a

term

vector

x

-LRB-

x

x

V

-RRB-

where

V

-LCB-

w

w

V

-RCB-

is

the

set

of

all

terms

-LRB-

i.e.

vocabulary

-RRB-

and

x

i

is

the

weight

of

term

w

i

Such

a

term

vector

is

commonly

used

in

information

retrieval

to

represent

both

queries

and

documents

For

example

the

vector-space

model

assumes

that

both

the

query

and

the

documents

are

represented

as

term

vectors

and

the

score

of

a

document

with

respect

to

a

query

is

computed

based

on

the

similarity

between

the

query

vector

and

the

document

vector

-LSB-

-RSB-

In

a

language

modeling

approach

we

may

also

regard

the

query

unigram

language

model

-LSB-

-RSB-

or

the

relevance

model

-LSB-

-RSB-

as

a

term

vector

representation

of

the

user

's

information

need

Intuitively

x

would

assign

high

weights

to

terms

that

characterize

the

topics

which

the

user

is

interested

in

The

second

component

we

may

include

in

our

user

model

is

the

documents

that

the

user

has

already

viewed

Obviously

even

if

a

document

is

relevant

if

the

user

has

already

seen

the

document

it

would

not

be

useful

to

present

the

same

document

again

We

thus

introduce

another

variable

S

D

-LRB-

D

is

the

whole

set

of

documents

in

the

collection

-RRB-

to

denote

the

subset

of

documents

in

the

search

results

that

the

user

has

already

seen/viewed

In

general

at

time

t

we

may

represent

a

user

model

as

m

t

-LRB-

S

x

A

t

R

-RRB-

where

S

is

the

seen

documents

x

is

the

system

's

understanding

of

the

user

's

information

need

and

-LRB-

A

t

R

-RRB-

represents

the

user

's

interaction

history

Note

that

an

even

more

general

user

model

may

also

include

other

factors

such

as

the

user

's

reading

level

and

occupation

If

we

assume

that

the

uncertainty

of

a

user

model

m

t

is

solely

due

to

the

uncertainty

of

x

the

computation

of

our

current

estimate

of

user

model

m

t

will

mainly

involve

computing

our

best

estimate

of

x

That

is

the

system

would

choose

a

response

according

to

r

t

argmin

rR

-LRB-

a

t

-RRB-

L

-LRB-

a

t

r

S

x

A

t

R

-RRB-

-LRB-

-RRB-

where

x

argmax

x

P

-LRB-

x

U

D

A

t

R

-RRB-

This

is

the

decision

mechanism

implemented

in

the

UCAIR

system

to

be

described

later

In

this

system

we

avoided

specifying

the

probabilistic

model

P

-LRB-

x

U

D

A

t

R

-RRB-

by

computing

x

directly

with

some

existing

feedback

method

Loss

functions

The

exact

definition

of

loss

function

L

depends

on

the

responses

thus

it

is

inevitably

application-specific

We

now

briefly

discuss

some

possibilities

when

the

response

is

to

rank

all

the

unseen

documents

and

present

the

top

k

of

them

Let

r

-LRB-

d

d

k

-RRB-

be

the

top

k

documents

S

be

the

set

of

seen

documents

by

the

user

and

x

be

the

system

's

best

guess

of

the

user

's

information

need

We

may

simply

define

the

loss

associated

with

r

as

the

negative

sum

of

the

probability

that

each

of

the

d

i

is

relevant

i.e.

L

-LRB-

a

r

m

-RRB-

k

i

P

-LRB-

relevant

d

i

m

-RRB-

Clearly

in

order

to

minimize

this

loss

function

the

optimal

response

r

would

contain

the

k

documents

with

the

highest

probability

of

relevance

which

is

intuitively

reasonable

One

deficiency

of

this

top-k

loss

function

is

that

it

is

not

sensitive

to

the

internal

order

of

the

selected

top

k

documents

so

switching

the

ranking

order

of

a

non-relevant

document

and

a

relevant

one

would

not

affect

the

loss

which

is

unreasonable

To

model

ranking

we

can

introduce

a

factor

of

the

user

model

the

probability

of

each

of

the

k

documents

being

viewed

by

the

user

P

-LRB-

view

d

i

-RRB-

and

define

the

following

ranking

loss

function

L

-LRB-

a

r

m

-RRB-

k

i

P

-LRB-

view

d

i

-RRB-

P

-LRB-

relevant

d

i

m

-RRB-

Since

in

general

if

d

i

is

ranked

above

d

j

-LRB-

i.e.

i

j

-RRB-

P

-LRB-

view

d

i

-RRB-

P

-LRB-

view

d

j

-RRB-

this

loss

function

would

favor

a

decision

to

rank

relevant

documents

above

non-relevant

ones

as

otherwise

we

could

always

switch

d

i

with

d

j

to

reduce

the

loss

value

Thus

the

system

should

simply

perform

a

regular

retrieval

and

rank

documents

according

to

the

probability

of

relevance

-LSB-

-RSB-

Depending

on

the

user

's

retrieval

preferences

there

can

be

many

other

possibilities

For

example

if

the

user

does

not

want

to

see

redundant

documents

the

loss

function

should

include

some

redundancy

measure

on

r

based

on

the

already

seen

documents

S.

Of

course

when

the

response

is

not

to

choose

a

ranked

list

of

documents

we

would

need

a

different

loss

function

We

discuss

one

such

example

that

is

relevant

to

the

search

agent

that

we

implement

When

a

user

enters

a

query

q

t

-LRB-

current

action

-RRB-

our

search

agent

relies

on

some

existing

search

engine

to

actually

carry

out

search

In

such

a

case

even

though

the

search

agent

does

not

have

control

of

the

retrieval

algorithm

it

can

still

attempt

to

optimize

the

search

results

through

refining

the

query

sent

to

the

search

engine

and/or

reranking

the

results

obtained

from

the

search

engine

The

loss

functions

for

reranking

are

already

discussed

above

we

now

take

a

look

at

the

loss

functions

for

query

refinement

Let

f

be

the

retrieval

function

of

the

search

engine

that

our

agent

uses

so

that

f

-LRB-

q

-RRB-

would

give

us

the

search

results

using

query

q.

Given

that

the

current

action

of

the

user

is

entering

a

query

q

t

-LRB-

i.e.

a

t

q

t

-RRB-

our

response

would

be

f

-LRB-

q

-RRB-

for

some

q

Since

we

have

no

choice

of

f

our

decision

is

to

choose

a

good

q.

Formally

r

t

argmin

r

t

L

-LRB-

a

r

t

m

-RRB-

argmin

f

-LRB-

q

-RRB-

L

-LRB-

a

f

-LRB-

q

-RRB-

m

-RRB-

f

-LRB-

argmin

q

L

-LRB-

q

t

f

-LRB-

q

-RRB-

m

-RRB-

-RRB-

which

shows

that

our

goal

is

to

find

q

argmin

q

L

-LRB-

q

t

f

-LRB-

q

-RRB-

m

-RRB-

i.e.

an

optimal

query

that

would

give

us

the

best

f

-LRB-

q

-RRB-

A

different

choice

of

loss

function

L

-LRB-

q

t

f

-LRB-

q

-RRB-

m

-RRB-

would

lead

to

a

different

query

refinement

strategy

In

UCAIR

we

heuristically

compute

q

by

expanding

q

t

with

terms

extracted

from

r

whenever

q

and

q

t

have

high

similarity

Note

that

r

and

q

are

contained

in

m

as

part

of

the

user

's

interaction

history

Implicit

user

modeling

Implicit

user

modeling

is

captured

in

our

framework

through

the

computation

of

x

argmax

x

P

-LRB-

x

U

D

A

t

R

-RRB-

i.e.

the

system

's

current

belief

of

what

the

user

's

information

need

is

Here

again

there

may

be

many

possibilities

leading

to

different

algorithms

for

implicit

user

modeling

We

now

discuss

a

few

of

them

First

when

two

consecutive

queries

are

related

the

previous

query

can

be

exploited

to

enrich

the

current

query

and

provide

more

search

context

to

help

disambiguation

For

this

purpose

instead

of

performing

query

expansion

as

we

did

in

the

previous

section

we

could

also

compute

an

updated

x

based

on

the

previous

query

and

retrieval

results

The

computed

new

user

model

can

then

be

used

to

rank

the

documents

with

a

standard

information

retrieval

model

Second

we

can

also

infer

a

user

's

interest

based

on

the

summaries

of

the

viewed

documents

When

a

user

is

presented

with

a

list

of

summaries

of

top

ranked

documents

if

the

user

chooses

to

skip

the

first

n

documents

and

to

view

the

-LRB-

n

-RRB-

th

document

we

may

infer

that

the

user

is

not

interested

in

the

displayed

summaries

for

the

first

n

documents

but

is

attracted

by

the

displayed

summary

of

the

-LRB-

n

-RRB-

th

document

We

can

thus

use

these

summaries

as

negative

and

positive

examples

to

learn

a

more

accurate

user

model

x

Here

many

standard

relevance

feedback

techniques

can

be

exploited

-LSB-

-RSB-

Note

that

we

should

use

the

displayed

summaries

as

opposed

to

the

actual

contents

of

those

documents

since

it

is

possible

that

the

displayed

summary

of

the

viewed

document

is

relevant

but

the

document

content

is

actually

not

Similarly

a

displayed

summary

may

mislead

a

user

to

skip

a

relevant

document

Inferring

user

models

based

on

such

displayed

information

rather

than

the

actual

content

of

a

document

is

an

important

difference

between

UCAIR

and

some

other

similar

systems

In

UCAIR

both

of

these

strategies

for

inferring

an

implicit

user

model

are

implemented

In

this

section

we

present

a

client-side

web

search

agent

called

UCAIR

in

which

we

implement

some

of

the

methods

discussed

in

the

previous

section

for

performing

personalized

search

through

implicit

user

modeling

UCAIR

is

a

web

browser

plug-in

that

acts

as

a

proxy

for

web

search

engines

Currently

it

is

only

implemented

for

Internet

Explorer

and

Google

but

it

is

a

matter

of

engineering

to

make

it

run

on

other

web

browsers

and

interact

with

other

search

engines

The

issue

of

privacy

is

a

primary

obstacle

for

deploying

any

real

world

applications

involving

serious

user

modeling

such

as

personalized

search

For

this

reason

UCAIR

is

strictly

running

as

a

client-side

search

agent

as

opposed

to

a

server-side

application

This

way

the

captured

user

information

always

resides

on

the

computer

that

the

user

is

using

thus

the

user

does

not

need

to

release

any

information

to

the

outside

Client-side

personalization

also

allows

the

system

to

easily

observe

a

lot

of

user

information

that

may

not

be

easily

available

to

a

server

Furthermore

performing

personalized

search

on

the

client-side

is

more

scalable

than

on

the

serverside

since

the

overhead

of

computation

and

storage

is

distributed

among

clients

As

shown

in

Figure

the

UCAIR

toolbar

has

major

components

-LRB-

-RRB-

The

-LRB-

implicit

-RRB-

user

modeling

module

captures

a

user

's

search

context

and

history

information

including

the

submitted

queries

and

any

clicked

search

results

and

infers

search

session

boundaries

-LRB-

-RRB-

The

query

modification

module

selectively

improves

the

query

formulation

according

to

the

current

user

model

-LRB-

-RRB-

The

result

re-ranking

module

immediately

re-ranks

any

unseen

search

results

whenever

the

user

model

is

updated

In

UCAIR

we

consider

four

basic

user

actions

-LRB-

-RRB-

submitting

a

keyword

query

-LRB-

-RRB-

viewing

a

document

-LRB-

-RRB-

clicking

the

Back

button

-LRB-

-RRB-

clicking

the

Next

link

on

a

result

page

For

each

of

these

four

actions

the

system

responds

with

respectively

-LRB-

-RRB-

UCAIR

is

available

at

Search

Engine

-LRB-

e.g.

Google

-RRB-

Search

History

Log

-LRB-

e.g.

past

queries

clicked

results

-RRB-

Query

Modification

Result

Re-Ranking

User

Modeling

Result

Buffer

UCAIR

User

query

results

clickthrough

Figure

UCAIR

architecture

generating

a

ranked

list

of

results

by

sending

a

possibly

expanded

query

to

a

search

engine

-LRB-

-RRB-

updating

the

information

need

model

x

-LRB-

-RRB-

reranking

the

unseen

results

on

the

current

result

page

based

on

the

current

model

x

and

-LRB-

-RRB-

reranking

the

unseen

pages

and

generating

the

next

page

of

results

based

on

the

current

model

x.

Behind

these

responses

there

are

three

basic

tasks

-LRB-

-RRB-

Decide

whether

the

previous

query

is

related

to

the

current

query

and

if

so

expand

the

current

query

with

useful

terms

from

the

previous

query

or

the

results

of

the

previous

query

-LRB-

-RRB-

Update

the

information

need

model

x

based

on

a

newly

clicked

document

summary

-LRB-

-RRB-

Rerank

a

set

of

unseen

documents

based

on

the

current

model

x.

Below

we

describe

our

algorithms

for

each

of

them

Session

boundary

detection

and

query

expansion

To

effectively

exploit

previous

queries

and

their

corresponding

clickthrough

information

UCAIR

needs

to

judge

whether

two

adjacent

queries

belong

to

the

same

search

session

-LRB-

i.e.

detect

session

boundaries

-RRB-

Existing

work

on

session

boundary

detection

is

mostly

in

the

context

of

web

log

analysis

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

and

uses

statistical

information

rather

than

textual

features

Since

our

client-side

agent

does

not

have

access

to

server

query

logs

we

make

session

boundary

decisions

based

on

textual

similarity

between

two

queries

Because

related

queries

do

not

necessarily

share

the

same

words

-LRB-

e.g.

java

island

and

travel

Indonesia

-RRB-

it

is

insufficient

to

use

only

query

text

Therefore

we

use

the

search

results

of

the

two

queries

to

help

decide

whether

they

are

topically

related

For

example

for

the

above

queries

java

island

and

travel

Indone-sia

the

words

java

bali

island

indonesia

and

travel

may

occur

frequently

in

both

queries

search

results

yielding

a

high

similarity

score

We

only

use

the

titles

and

summaries

of

the

search

results

to

calculate

the

similarity

since

they

are

available

in

the

retrieved

search

result

page

and

fetching

the

full

text

of

every

result

page

would

sig-nificantly

slow

down

the

process

To

compensate

for

the

terseness

of

titles

and

summaries

we

retrieve

more

results

than

a

user

would

normally

view

for

the

purpose

of

detecting

session

boundaries

-LRB-

typ-ically

results

-RRB-

The

similarity

between

the

previous

query

q

and

the

current

query

q

is

computed

as

follows

Let

-LCB-

s

s

s

n

-RCB-

and

-LCB-

s

s

s

n

-RCB-

be

the

result

sets

for

the

two

queries

We

use

the

pivoted

normalization

TF-IDF

weighting

formula

-LSB-

-RSB-

to

compute

a

term

weight

vector

s

i

for

each

result

s

i

We

define

the

average

result

s

avg

to

be

the

centroid

of

all

the

result

vectors

i.e.

-LRB-

s

s

s

n

-RRB-

n

The

cosine

similarity

between

the

two

average

results

is

calculated

as

s

avg

s

avg

s

avg

s

avg

If

the

similarity

value

exceeds

a

predefined

threshold

the

two

queries

will

be

considered

to

be

in

the

same

information

session

If

the

previous

query

and

the

current

query

are

found

to

belong

to

the

same

search

session

UCAIR

would

attempt

to

expand

the

current

query

with

terms

from

the

previous

query

and

its

search

results

Specifically

for

each

term

in

the

previous

query

or

the

corresponding

search

results

if

its

frequency

in

the

results

of

the

current

query

is

greater

than

a

preset

threshold

-LRB-

e.g.

results

out

of

-RRB-

the

term

would

be

added

to

the

current

query

to

form

an

expanded

query

In

this

case

UCAIR

would

send

this

expanded

query

rather

than

the

original

one

to

the

search

engine

and

return

the

results

corresponding

to

the

expanded

query

Currently

UCAIR

only

uses

the

immediate

preceding

query

for

query

expansion

in

principle

we

could

exploit

all

related

past

queries

Information

need

model

updating

Suppose

at

time

t

we

have

observed

that

the

user

has

viewed

k

documents

whose

summaries

are

s

s

k

We

update

our

user

model

by

computing

a

new

information

need

vector

with

a

standard

feedback

method

in

information

retrieval

-LRB-

i.e.

Rocchio

-LSB-

-RSB-

-RRB-

According

to

the

vector

space

retrieval

model

each

clicked

summary

s

i

can

be

represented

by

a

term

weight

vector

s

i

with

each

term

weighted

by

a

TF-IDF

weighting

formula

-LSB-

-RSB-

Rocchio

computes

the

centroid

vector

of

all

the

summaries

and

interpolates

it

with

the

original

query

vector

to

obtain

an

updated

term

vector

That

is

x

q

-LRB-

-RRB-

k

k

i

s

i

where

q

is

the

query

vector

k

is

the

number

of

summaries

the

user

clicks

immediately

following

the

current

query

and

is

a

parameter

that

controls

the

influence

of

the

clicked

summaries

on

the

inferred

information

need

model

In

our

experiments

is

set

to

Note

that

we

update

the

information

need

model

whenever

the

user

views

a

document

Result

reranking

In

general

we

want

to

rerank

all

the

unseen

results

as

soon

as

the

user

model

is

updated

Currently

UCAIR

implements

reranking

in

two

cases

corresponding

to

the

user

clicking

the

Back

button

and

Next

link

in

the

Internet

Explorer

In

both

cases

the

current

-LRB-

updated

-RRB-

user

model

would

be

used

to

rerank

the

unseen

results

so

that

the

user

would

see

improved

search

results

immediately

To

rerank

any

unseen

document

summaries

UCAIR

uses

the

standard

vector

space

retrieval

model

and

scores

each

summary

based

on

the

similarity

of

the

result

and

the

current

user

information

need

vector

x

-LSB-

-RSB-

Since

implicit

feedback

is

not

completely

reliable

we

bring

up

only

a

small

number

-LRB-

e.g.

-RRB-

of

highest

reranked

results

to

be

followed

by

any

originally

high

ranked

results

Google

result

-LRB-

user

query

java

map

-RRB-

UCAIR

result

-LRB-

user

query

java

map

-RRB-

previous

query

travel

Indonesia

previous

query

hashtable

expanded

user

query

java

map

Indonesia

expanded

user

query

java

map

class

Java

map

projections

of

the

world

Lonely

Planet

Indonesia

Map

Map

-LRB-

Java

Platform

SE

-RRB-

Java

map

projections

of

the

world

INDONESIA

TOURISM

CENTRAL

JAVA

MAP

Java

Platform

SE

Interface

Map

Java

Map

INDONESIA

TOURISM

WEST

JAVA

MAP

An

Introduction

to

Java

Map

Collection

Classes

Java

Technology

Concept

Map

IndoStreets

Java

Map

An

Introduction

to

Java

Map

Collection

Classes

Science@NASA

Home

Indonesia

Regions

and

Islands

Maps

Bali

Java

Koders

Mappings.java

science.nasa.gov

Realtime

An

Introduction

to

Java

Map

Collection

Classes

Indonesia

City

Street

Map

Hibernate

simplifies

inheritance

mapping

Lonely

Planet

Java

Map

Maps

Of

Indonesia

tmap

map

Class

Hierarchy

tmap.pmel.noaa.gov

ONJava.com

Java

API

Map

Maps

of

Indonesia

by

Peter

Loud

Class

Scope

map

users.powernet.co.uk

GTA

San

Andreas

Sam

Maps

of

Indonesia

by

Peter

Loud

Class

PrintSafeHashMap

users.powernet.co.uk

mkmarina/indonesia

INDONESIA

TOURISM

WEST

JAVA

MAP

indonesiaphoto.com

Java

Pro

Union

and

Vertical

Mapping

of

Classes

Table

Sample

results

of

query

expansion

We

now

present

some

results

on

evaluating

the

two

major

UCAIR

functions

selective

query

expansion

and

result

reranking

based

on

user

clickthrough

data

Sample

results

The

query

expansion

strategy

implemented

in

UCAIR

is

inten-tionally

conservative

to

avoid

misinterpretation

of

implicit

user

models

In

practice

whenever

it

chooses

to

expand

the

query

the

expansion

usually

makes

sense

In

Table

we

show

how

UCAIR

can

successfully

distinguish

two

different

search

contexts

for

the

query

java

map

corresponding

to

two

different

previous

queries

-LRB-

i.e.

travel

Indonesia

vs.

hashtable

-RRB-

Due

to

implicit

user

modeling

UCAIR

intelligently

figures

out

to

add

Indonesia

and

class

respectively

to

the

user

's

query

java

map

which

would

otherwise

be

ambiguous

as

shown

in

the

original

results

from

Google

on

March

UCAIR

's

results

are

much

more

accurate

than

Google

's

results

and

reflect

personalization

in

search

The

eager

implicit

feedback

component

is

designed

to

immediately

respond

to

a

user

's

activity

such

as

viewing

a

document

In

Figure

we

show

how

UCAIR

can

successfully

disambiguate

an

ambiguous

query

jaguar

by

exploiting

a

viewed

document

summary

In

this

case

the

initial

retrieval

results

using

jaguar

-LRB-

shown

on

the

left

side

-RRB-

contain

two

results

about

the

Jaguar

cars

followed

by

two

results

about

the

Jaguar

software

However

after

the

user

views

the

web

page

content

of

the

second

result

-LRB-

about

Jaguar

car

-RRB-

and

returns

to

the

search

result

page

by

clicking

Back

button

UCAIR

automatically

nominates

two

new

search

results

about

Jaguar

cars

-LRB-

shown

on

the

right

side

-RRB-

while

the

original

two

results

about

Jaguar

software

are

pushed

down

on

the

list

-LRB-

unseen

from

the

picture

-RRB-

Quantitative

evaluation

To

further

evaluate

UCAIR

quantitatively

we

conduct

a

user

study

on

the

effectiveness

of

the

eager

implicit

feedback

component

It

is

a

challenge

to

quantitatively

evaluate

the

potential

performance

improvement

of

our

proposed

model

and

UCAIR

over

Google

in

an

unbiased

way

-LSB-

-RSB-

Here

we

design

a

user

study

in

which

participants

would

do

normal

web

search

and

judge

a

randomly

and

anonymously

mixed

set

of

results

from

Google

and

UCAIR

at

the

end

of

the

search

session

participants

do

not

know

whether

a

result

comes

from

Google

or

UCAIR

We

recruited

graduate

students

for

this

user

study

who

have

different

backgrounds

-LRB-

computer

science

biology

and

chem

lt

top

num

Number

title

Spammer

arrest

sue

desc

Description

Have

any

spammers

been

arrested

or

sued

for

sending

unsolicited

e-mail

narr

Narrative

Instances

of

arrests

prosecutions

convictions

and

punishments

of

spammers

and

lawsuits

against

them

are

relevant

Documents

which

describe

laws

to

limit

spam

without

giving

details

of

lawsuits

or

criminal

trials

are

not

relevant

top

Figure

An

example

of

TREC

query

topic

expressed

in

a

form

which

might

be

given

to

a

human

assistant

or

librarian

istry

-RRB-

We

use

query

topics

from

TREC

Terabyte

track

-LSB-

-RSB-

and

TREC

Web

track

-LSB-

-RSB-

topic

distillation

task

in

the

way

to

be

described

below

An

example

topic

from

TREC

Terabyte

track

appears

in

Figure

The

title

is

a

short

phrase

and

may

be

used

as

a

query

to

the

retrieval

system

The

description

field

provides

a

slightly

longer

statement

of

the

topic

requirement

usually

expressed

as

a

single

complete

sentence

or

question

Finally

the

narrative

supplies

additional

information

necessary

to

fully

specify

the

requirement

expressed

in

the

form

of

a

short

paragraph

Initially

each

participant

would

browse

topics

either

from

Terabyte

track

or

Web

track

and

pick

or

most

interesting

topics

For

each

picked

topic

the

participant

would

essentially

do

the

normal

web

search

using

UCAIR

to

find

many

relevant

web

pages

by

using

the

title

of

the

query

topic

as

the

initial

keyword

query

During

this

process

the

participant

may

view

the

search

results

and

possibly

click

on

some

interesting

ones

to

view

the

web

pages

just

as

in

a

normal

web

search

There

is

no

requirement

or

restriction

on

how

many

queries

the

participant

must

submit

or

when

the

participant

should

stop

the

search

for

one

topic

When

the

participant

plans

to

change

the

search

topic

he/she

will

simply

press

a

button

Text

REtrieval

Conference

Figure

Screen

shots

for

result

reranking

to

evaluate

the

search

results

before

actually

switching

to

the

next

topic

At

the

time

of

evaluation

top

ranked

results

from

Google

and

UCAIR

-LRB-

some

are

overlapping

-RRB-

are

randomly

mixed

together

so

that

the

participant

would

not

know

whether

a

result

comes

from

Google

or

UCAIR

The

participant

would

then

judge

the

relevance

of

these

results

We

measure

precision

at

top

n

-LRB-

n

-RRB-

documents

of

Google

and

UCAIR

We

also

evaluate

precisions

at

different

recall

levels

Altogether

documents

judged

as

relevant

from

Google

search

results

and

documents

judged

as

relevant

from

UCAIR

by

participants

Scatter

plots

of

precision

at

top

and

top

documents

are

shown

in

Figure

and

Figure

respectively

-LRB-

The

scatter

plot

of

precision

at

top

documents

is

very

similar

to

precision

at

top

documents

-RRB-

Each

point

of

the

scatter

plots

represents

the

precisions

of

Google

and

UCAIR

on

one

query

topic

Table

shows

the

average

precision

at

top

n

documents

among

topics

From

Figure

Figure

and

Table

we

see

that

the

search

results

from

UCAIR

are

consistently

better

than

those

from

Google

by

all

the

measures

Moreover

the

performance

improvement

is

more

dramatic

for

precision

at

top

documents

than

that

at

precision

at

top

documents

One

explanation

for

this

is

that

the

more

interaction

the

user

has

with

the

system

the

more

clickthrough

data

UCAIR

can

be

expected

to

collect

Thus

the

retrieval

system

can

build

more

precise

implicit

user

models

which

lead

to

better

retrieval

accuracy

Ranking

Method

prec

Google

UCAIR

Improvement

Table

Table

of

average

precision

at

top

n

documents

for

query

topics

The

plot

in

Figure

shows

the

precision-recall

curves

for

UCAIR

and

Google

where

it

is

clearly

seen

that

the

performance

of

UCAIR

UCAIR

Google

Scatterplot

of

Precision

at

Top

Documents

Figure

Precision

at

top

documents

of

UCAIR

and

Google

is

consistently

and

considerably

better

than

that

of

Google

at

all

levels

of

recall

In

this

paper

we

studied

how

to

exploit

implicit

user

modeling

to

intelligently

personalize

information

retrieval

and

improve

search

accuracy

Unlike

most

previous

work

we

emphasize

the

use

of

immediate

search

context

and

implicit

feedback

information

as

well

as

eager

updating

of

search

results

to

maximally

benefit

a

user

We

presented

a

decision-theoretic

framework

for

optimizing

interactive

information

retrieval

based

on

eager

user

model

updating

in

which

the

system

responds

to

every

action

of

the

user

by

choosing

a

system

action

to

optimize

a

utility

function

We

further

propose

specific

techniques

to

capture

and

exploit

two

types

of

implicit

feedback

information

-LRB-

-RRB-

identifying

related

immediately

preceding

query

and

using

the

query

and

the

corresponding

search

results

to

select

appropriate

terms

to

expand

the

current

query

and

-LRB-

-RRB-

exploiting

the

viewed

document

summaries

to

immediately

rerank

any

documents

that

have

not

yet

been

seen

by

the

user

Using

these

techniques

we

develop

a

client-side

web

search

agent

-LRB-

UCAIR

-RRB-

on

top

of

a

popular

search

engine

-LRB-

Google

-RRB-

Experiments

on

web

search

show

that

our

search

agent

can

improve

search

accuracy

over

UCAIR

Google

Scatterplot

of

Precision

at

Top

documents

Figure

Precision

at

top

documents

of

UCAIR

and

Google

recall

precision

Precision-Recall

curves

Google

Result

UCAIR

Result

Figure

Precision

at

top

result

of

UCAIR

and

Google

Google

Since

the

implicit

information

we

exploit

already

naturally

exists

through

user

interactions

the

user

does

not

need

to

make

any

extra

effort

The

developed

search

agent

thus

can

improve

existing

web

search

performance

without

any

additional

effort

from

the

user

We

thank

the

six

participants

of

our

evaluation

experiments

This

work

was

supported

in

part

by

the

National

Science

Foundation

grants

and

-LSB-

-RSB-

S.

M.

Beitzel

E.

C.

Jensen

A.

Chowdhury

D.

Grossman

and

O.

Frieder

Hourly

analysis

of

a

very

large

topically

categorized

web

query

log

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

C.

Clarke

N.

Craswell

and

I.

Soboroff

Overview

of

the

TREC

terabyte

track

In

Proceedings

of

TREC

-LSB-

-RSB-

M.

Claypool

P.

Le

M.

Waseda

and

D.

Brown

Implicit

interest

indicators

In

Proceedings

of

Intelligent

User

Interfaces

pages

-LSB-

-RSB-

N.

Craswell

D.

Hawking

R.

Wilkinson

and

M.

Wu

Overview

of

the

TREC

web

track

In

Proceedings

of

TREC

-LSB-

-RSB-

W.

B.

Croft

S.

Cronen-Townsend

and

V.

Larvrenko

Relevance

feedback

and

personalization

A

language

modeling

perspective

In

Proeedings

of

Second

DELOS

Workshop

Personalisation

and

Recommender

Systems

in

Digital

Libraries

-LSB-

-RSB-

Google

Personalized

-LSB-

-RSB-

D.

Hawking

N.

Craswell

P.

B.

Thistlewaite

and

D.

Harman

Results

and

challenges

in

web

search

evaluation

Computer

Networks

-LRB-

-RRB-

-LSB-

-RSB-

X.

Huang

F.

Peng

A.

An

and

D.

Schuurmans

Dynamic

web

log

session

identification

with

statistical

language

models

Journal

of

the

American

Society

for

Information

Science

and

Technology

-LRB-

-RRB-

-LSB-

-RSB-

G.

Jeh

and

J.

Widom

Scaling

personalized

web

search

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

T.

Joachims

Optimizing

search

engines

using

clickthrough

data

In

Proceedings

of

SIGKDD

pages

-LSB-

-RSB-

D.

Kelly

and

J.

Teevan

Implicit

feedback

for

inferring

user

preference

A

bibliography

SIGIR

Forum

-LRB-

-RRB-

-LSB-

-RSB-

J.

Lafferty

and

C.

Zhai

Document

language

models

query

models

and

risk

minimization

for

information

retrieval

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

T.

Lau

and

E.

Horvitz

Patterns

of

search

Analyzing

and

modeling

web

query

refinement

In

Proceedings

of

the

Seventh

International

Conference

on

User

Modeling

-LRB-

UM

-RRB-

pages

-LSB-

-RSB-

V.

Lavrenko

and

B.

Croft

Relevance-based

language

models

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

M.

Mitra

A.

Singhal

and

C.

Buckley

Improving

automatic

query

expansion

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

My

Yahoo!

-LSB-

-RSB-

G.

Nunberg

As

google

goes

so

goes

the

nation

New

York

Times

May

-LSB-

-RSB-

S.

E.

Robertson

The

probability

ranking

principle

in

i

Journal

of

Documentation

-LRB-

-RRB-

-LSB-

-RSB-

J.

J.

Rocchio

Relevance

feedback

in

information

retrieval

In

The

SMART

Retrieval

System

Experiments

in

Automatic

Document

Processing

pages

Prentice-Hall

Inc.

-LSB-

-RSB-

G.

Salton

and

C.

Buckley

Improving

retrieval

performance

by

retrieval

feedback

Journal

of

the

American

Society

for

Information

Science

-LRB-

-RRB-

-LSB-

-RSB-

G.

Salton

and

M.

J.

McGill

Introduction

to

Modern

Information

Retrieval

McGraw-Hill

-LSB-

-RSB-

X.

Shen

B.

Tan

and

C.

Zhai

Context-sensitive

information

retrieval

using

implicit

feedback

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

X.

Shen

and

C.

Zhai

Exploiting

query

history

for

document

ranking

in

interactive

information

retrieval

-LRB-

Poster

-RRB-

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

A.

Singhal

Modern

information

retrieval

A

brief

overview

Bulletin

of

the

IEEE

Computer

Society

Technical

Committee

on

Data

Engineering

-LRB-

-RRB-

-LSB-

-RSB-

K.

Sugiyama

K.

Hatano

and

M.

Yoshikawa

Adaptive

web

search

based

on

user

profile

constructed

without

any

effort

from

users

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

E.

Volokh

Personalization

and

privacy

Communications

of

the

ACM

-LRB-

-RRB-

-LSB-

-RSB-

R.

W.

White

J.

M.

Jose

C.

J.

van

Rijsbergen

and

I.

Ruthven

A

simulated

study

of

implicit

feedback

models

In

Proceedings

of

ECIR

pages

-LSB-

-RSB-

J.

Xu

and

W.

B.

Croft

Query

expansion

using

local

and

global

document

analysis

In

Proceedings

of

SIGIR

pages

-LSB-

-RSB-

C.

Zhai

and

J.

Lafferty

Model-based

feedback

in

KL

divergence

retrieval

model

In

Proceedings

of

the

CIKM

pages

O O O O B I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

retrieval accuracy
implicit feedback
user model
clickthrough information
ucair
information retrieval systems
personalized search
interactive retrieval