Probabilistic

Author-Topic

Models

for

Information

Discovery

We

propose

a

new

unsupervised

learning

technique

for

extracting

information

from

large

text

collections

We

model

documents

as

if

they

were

generated

by

a

two-stage

stochastic

process

Each

author

is

represented

by

a

probability

distribution

over

topics

and

each

topic

is

represented

as

a

probability

distribution

over

words

for

that

topic

The

words

in

a

multi-author

paper

are

assumed

to

be

the

result

of

a

mixture

of

each

authors

topic

mixture

The

topic-word

and

author-topic

distributions

are

learned

from

data

in

an

unsupervised

manner

using

a

Markov

chain

Monte

Carlo

algorithm

We

apply

the

methodology

to

a

large

corpus

of

abstracts

and

authors

from

the

well-known

CiteSeer

digital

library

and

learn

a

model

with

topics

We

discuss

in

detail

the

interpretation

of

the

results

discovered

by

the

system

including

specific

topic

and

author

models

ranking

of

authors

by

topic

and

topics

by

author

significant

trends

in

the

computer

science

literature

between

and

parsing

of

abstracts

by

topics

and

authors

and

detection

of

unusual

papers

by

specific

authors

An

online

query

interface

to

the

model

is

also

discussed

that

allows

interactive

exploration

of

author-topic

models

for

corpora

such

as

CiteSeer

I.

-LSB-

Artificial

Intelligence

-RSB-

Learning

algorithms

experimentation

With

the

advent

of

the

Web

and

various

specialized

digital

libraries

the

automatic

extraction

of

useful

information

from

text

has

become

an

increasingly

important

research

area

in

data

mining

In

this

paper

we

discuss

a

new

algo-Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

KDD

August

Seattle

Washington

USA

Copyright

ACM

rithm

that

extracts

both

the

topics

expressed

in

large

text

document

collections

and

models

how

the

authors

of

documents

use

those

topics

The

methodology

is

illustrated

using

a

sample

of

abstracts

and

authors

from

the

well-known

CiteSeer

digital

library

of

computer

science

research

papers

-LRB-

Lawrence

Giles

and

Bollacker

-RRB-

The

algorithm

uses

a

probabilistic

model

that

represents

topics

as

probability

distributions

over

words

and

documents

as

being

composed

of

multiple

topics

A

novel

feature

of

our

model

is

the

inclusion

of

author

models

in

which

authors

are

modeled

as

probability

distributions

over

topics

The

author-topic

models

can

be

used

to

support

a

variety

of

interactive

and

exploratory

queries

on

the

set

of

documents

and

authors

including

analysis

of

topic

trends

over

time

finding

the

authors

who

are

most

likely

to

write

on

a

given

topic

and

finding

the

most

unusual

paper

written

by

a

given

author

Bayesian

unsupervised

learning

is

used

to

fit

the

model

to

a

document

collection

Supervised

learning

techniques

for

automated

categorization

of

documents

into

known

classes

or

topics

has

received

considerable

attention

in

recent

years

-LRB-

e.g.

Yang

-RRB-

For

many

document

collections

however

neither

predefined

topics

nor

labeled

documents

may

be

available

Furthermore

there

is

considerable

motivation

to

uncover

hidden

topic

structure

in

large

corpora

particularly

in

rapidly

changing

fields

such

as

computer

science

and

biology

where

predefined

topic

categories

may

not

accurately

reflect

rapidly

evolving

content

Automatic

extraction

of

topics

from

text

via

unsupervised

learning

has

been

addressed

in

prior

work

using

a

number

of

different

approaches

One

general

approach

is

to

represent

the

high-dimensional

term

vectors

in

a

lower-dimensional

space

Local

regions

in

the

lower-dimensional

space

can

then

be

associated

with

specific

topics

For

example

the

WEBSOM

system

-LRB-

Lagus

et

al.

-RRB-

uses

nonlinear

dimensionality

reduction

via

self-organizing

maps

to

represent

term

vectors

in

a

two-dimensional

layout

Linear

projection

techniques

such

as

latent

semantic

indexing

-LRB-

LSI

-RRB-

are

also

widely

used

-LRB-

Berry

Dumais

and

O

Brien

-RRB-

For

example

Deerwester

et

al.

-LRB-

-RRB-

while

not

using

the

term

topics

per

se

state

Roughly

speaking

these

factors

may

be

thought

of

as

artificial

concepts

they

represent

extracted

common

meaning

components

of

many

different

words

and

documents

Research

Track

Paper

A

somewhat

different

approach

is

to

cluster

the

documents

into

groups

containing

similar

semantic

content

using

any

of

a

variety

of

well-known

document

clustering

techniques

-LRB-

e.g.

Cutting

et

al.

McCallum

Nigam

and

Ungar

Popescul

et

al.

-RRB-

Each

cluster

of

documents

can

then

be

associated

with

a

latent

topic

-LRB-

e.g.

as

represented

by

the

mean

term

vector

for

documents

in

the

cluster

-RRB-

While

clustering

can

provide

useful

broad

information

about

topics

clusters

are

inherently

limited

by

the

fact

that

each

document

is

-LRB-

typically

-RRB-

only

associated

with

one

cluster

This

is

often

at

odds

with

the

multi-topic

nature

of

text

documents

in

many

contexts

In

particular

combinations

of

diverse

topics

within

a

single

document

are

difficult

to

represent

For

example

this

present

paper

contains

at

least

two

significantly

different

topics

document

modeling

and

Bayesian

estimation

For

this

reason

other

representations

-LRB-

such

as

those

discussed

below

-RRB-

that

allow

documents

to

be

composed

of

multiple

topics

generally

provide

better

models

for

sets

of

documents

-LRB-

e.g.

better

out

of

sample

predictions

Blei

Ng

and

Jordan

-LRB-

-RRB-

-RRB-

Hofmann

-LRB-

-RRB-

introduced

the

aspect

model

-LRB-

also

referred

to

as

probabilistic

LSI

or

pLSI

-RRB-

as

a

probabilistic

alternative

to

projection

and

clustering

methods

In

pLSI

topics

are

modeled

as

multinomial

probability

distributions

over

words

and

documents

are

assumed

to

be

generated

by

the

activation

of

multiple

topics

While

the

pLSI

model

produced

impressive

results

on

a

number

of

text

document

problems

such

as

information

retrieval

the

parameterization

of

the

model

was

susceptible

to

overfitting

and

did

not

provide

a

straightforward

way

to

make

inferences

about

new

documents

not

seen

in

the

training

data

Blei

Ng

and

Jordan

-LRB-

-RRB-

addressed

these

limitations

by

proposing

a

more

general

Bayesian

probabilistic

topic

model

called

latent

Dirichlet

allocation

-LRB-

LDA

-RRB-

The

parameters

of

the

LDA

model

-LRB-

the

topic-word

and

document-topic

distributions

-RRB-

are

estimated

using

an

approximation

technique

known

as

variational

EM

since

standard

estimation

methods

are

intractable

Griffiths

and

Steyvers

-LRB-

-RRB-

showed

how

Gibbs

sampling

a

Markov

chain

Monte

Carlo

technique

could

be

applied

in

this

model

and

illustrated

this

approach

using

years

of

abstract

data

from

the

Proceedings

of

the

National

Academy

of

Sciences

Our

focus

here

is

to

extend

the

probabilistic

topic

models

to

include

authorship

information

Joint

author-topic

modeling

has

received

little

or

no

attention

as

far

as

we

are

aware

The

areas

of

stylometry

authorship

attribution

and

forensic

linguistics

focus

on

the

problem

of

identifying

what

author

wrote

a

given

piece

of

text

For

example

Mosteller

and

Wallace

-LRB-

-RRB-

used

Bayesian

techniques

to

infer

whether

Hamilton

or

Madison

was

the

more

likely

author

of

disputed

Federalist

papers

More

recent

work

of

a

similar

nature

includes

authorship

analysis

of

a

purported

poem

by

Shakespeare

-LRB-

Thisted

and

Efron

-RRB-

identifying

authors

of

software

programs

-LRB-

Gray

Sallis

and

MacDonell

-RRB-

and

the

use

of

techniques

such

as

support

vector

machines

-LRB-

Diederich

et

al.

-RRB-

for

author

identification

These

author

identification

methods

emphasize

the

use

of

distinctive

stylistic

features

-LRB-

such

as

sentence

length

-RRB-

that

characterize

a

specific

author

In

contrast

the

models

we

present

here

focus

on

extracting

the

general

semantic

content

of

a

document

rather

than

the

stylistic

details

of

how

it

was

written

For

example

in

our

model

we

omit

common

stop

words

since

they

are

generally

irrelevant

to

the

topic

of

the

document

however

the

distributions

of

stop

words

can

be

quite

useful

in

stylometry

While

topic

information

could

be

usefully

combined

with

stylistic

features

for

author

classification

we

do

not

pursue

this

idea

in

this

particular

paper

Graph-based

and

network-based

models

are

also

frequently

used

as

a

basis

for

representation

and

analysis

of

relations

among

scientific

authors

For

example

Newman

-LRB-

-RRB-

Mutschke

-LRB-

-RRB-

and

Erten

et

al.

-LRB-

-RRB-

use

methods

from

bibliometrics

social

networks

and

graph

theory

to

analyze

and

visualize

co-author

and

citation

relations

in

the

scientific

literature

Kautz

Selman

and

Shah

-LRB-

-RRB-

developed

the

interactive

ReferralWeb

system

for

exploring

networks

of

computer

scientists

working

in

artificial

intelligence

and

information

retrieval

and

White

and

Smyth

-LRB-

-RRB-

used

PageRank-style

ranking

algorithms

to

analyze

co-author

graphs

In

all

of

this

work

only

the

network

con-nectivity

information

is

used

the

text

information

from

the

underlying

documents

is

not

used

in

modeling

Thus

while

the

grouping

of

authors

via

these

network

models

can

implicitly

provide

indications

of

latent

topics

there

is

no

explicit

representation

of

the

topics

in

terms

of

the

text

content

-LRB-

the

words

-RRB-

of

the

documents

The

novelty

of

the

work

described

in

this

paper

lies

in

the

proposal

of

a

probabilistic

model

that

represents

both

authors

and

topics

and

the

application

of

this

model

to

a

large

well-known

document

corpus

in

computer

science

As

we

will

show

later

in

the

paper

the

model

provides

a

general

framework

for

exploration

discovery

and

query-answering

in

the

context

of

the

relationships

of

author

and

topics

for

large

document

collections

The

outline

of

the

paper

is

as

follows

in

Section

we

describe

the

author-topic

model

and

outline

how

the

parameters

of

the

model

-LRB-

the

topic-word

distributions

and

author-topic

distributions

-RRB-

can

be

learned

from

training

data

consisting

of

documents

with

known

authors

Section

illustrates

the

application

of

the

model

to

a

large

collection

of

abstracts

from

the

CiteSeer

system

with

examples

of

specific

topics

and

specific

author

models

that

are

learned

by

the

algorithm

In

Section

we

illustrate

a

number

of

applications

of

the

model

including

the

characterization

of

topic

trends

over

time

-LRB-

which

provides

some

interesting

insights

on

the

direction

of

research

in

computer

science

-RRB-

and

the

characterization

of

which

papers

are

most

typical

and

least

typical

for

a

given

author

An

online

query

interface

to

the

system

is

described

in

Section

allowing

users

to

query

the

model

over

the

Web

an

interesting

feature

of

the

model

is

the

coupling

of

Bayesian

sampling

and

relational

database

technology

to

answer

queries

in

real-time

Section

contains

a

brief

discussion

of

future

directions

and

concluding

comments

The

author-topic

model

reduces

the

process

of

writing

a

scientific

document

to

a

simple

series

of

probabilistic

steps

The

model

not

only

discovers

what

topics

are

expressed

in

a

document

but

also

which

authors

are

associated

with

each

topic

To

simplify

the

representation

of

documents

we

use

a

bag

of

words

assumption

that

reduces

each

document

to

a

Research

Track

Paper

x

z

w

D

K

T

d

a

Given

the

set

of

co-authors

N

d

Choose

an

author

Choose

a

topic

given

the

author

Choose

a

word

given

the

topic

Figure

The

graphical

model

for

the

author-topic

model

using

plate

notation

vector

of

counts

where

each

vector

element

corresponds

to

the

number

of

times

a

term

appears

in

the

document

Each

author

is

associated

with

a

multinomial

distribution

over

topics

A

document

with

multiple

authors

has

a

distribution

over

topics

that

is

a

mixture

of

the

distributions

associated

with

the

authors

When

generating

a

document

an

author

is

chosen

at

random

for

each

individual

word

in

the

document

This

author

picks

a

topic

from

his

or

her

multinomial

distribution

over

topics

and

then

samples

a

word

from

the

multinomial

distribution

over

words

associated

with

that

topic

This

process

is

repeated

for

all

words

in

the

document

In

the

model

the

authors

produce

words

from

a

set

of

T

topics

When

T

is

kept

relatively

small

relative

to

the

number

of

authors

and

vocabulary

size

the

author-topic

model

applies

a

form

of

dimensionality

reduction

to

documents

topics

are

learned

which

capture

the

variability

in

word

choice

across

a

large

set

of

documents

and

authors

In

our

simulations

we

use

topics

-LRB-

see

Rosen-Zvi

et

al.

-LRB-

-RRB-

for

an

exploration

of

different

numbers

of

topics

-RRB-

Figure

illustrates

the

generative

process

with

a

graphical

model

using

plate

notation

For

readers

not

familiar

with

plate

notation

shaded

and

unshaded

variables

indicate

observed

and

latent

variables

respectively

An

arrow

indicates

a

conditional

dependency

between

variables

and

plates

-LRB-

the

boxes

in

the

figure

-RRB-

indicate

repeated

sampling

with

the

number

of

repetitions

given

by

the

variable

in

the

bottom

-LRB-

see

Buntine

-LRB-

-RRB-

for

an

introduction

-RRB-

In

the

author-topic

model

observed

variables

not

only

include

the

words

w

in

a

document

but

also

the

set

of

coauthors

A

d

on

each

document

d.

Currently

the

model

does

not

specify

the

generative

process

of

how

authors

choose

to

collaborate

Instead

we

assume

the

model

is

provided

with

the

authorship

information

on

every

document

in

the

collection

Each

author

-LRB-

from

a

set

of

K

authors

-RRB-

is

associated

with

a

multinomial

distribution

over

topics

represented

by

Each

topic

is

associated

with

a

multinomial

distribution

over

words

represented

by

The

multinomial

distributions

and

have

a

symmetric

Dirichlet

prior

with

hyperparame-ters

and

-LRB-

see

Rosen-Zvi

et

al.

-LRB-

-RRB-

for

details

-RRB-

For

each

word

in

the

document

we

sample

an

author

x

uni-formly

from

A

d

then

sample

a

topic

z

from

the

multinomial

distribution

associated

with

author

x

and

sample

a

word

w

from

a

multinomial

topic

distribution

associated

with

topic

z

This

sampling

process

is

repeated

N

times

to

form

document

d.

Bayesian

Estimation

of

the

Model

Parameters

The

author-topic

model

includes

two

sets

of

unknown

parameters

the

K

author-topic

distributions

and

the

T

topic

distributions

as

well

as

the

latent

variables

corresponding

to

the

assignments

of

individual

words

to

topics

z

and

authors

x

The

Expectation-Maximization

-LRB-

EM

-RRB-

algorithm

is

a

standard

technique

for

estimating

parameters

in

models

with

latent

variables

finding

a

mode

of

the

posterior

distribution

over

parameters

However

when

applied

to

probabilistic

topic

models

-LRB-

Hofmann

-RRB-

this

approach

is

susceptible

to

local

maxima

and

computationally

inefficient

-LRB-

see

Blei

Ng

and

Jordan

-RRB-

We

pursue

an

alternative

parameter

estimation

strategy

outlined

by

Griffiths

and

Steyvers

-LRB-

-RRB-

using

Gibbs

sampling

a

Markov

chain

Monte

Carlo

algorithm

to

sample

from

the

posterior

distribution

over

parameters

Instead

of

estimating

the

model

parameters

directly

we

evaluate

the

posterior

distribution

on

just

x

and

z

and

then

use

the

results

to

infer

and

For

each

word

the

topic

and

author

assignment

are

sam-pled

from

P

-LRB-

z

i

j

x

i

k

w

i

m

z

i

x

i

-RRB-

C

W

T

mj

m

C

W

T

m

j

V

C

AT

kj

j

C

AT

kj

T

-LRB-

-RRB-

where

z

i

j

and

x

i

k

represent

the

assignments

of

the

ith

word

in

a

document

to

topic

j

and

author

k

respectively

w

i

m

represents

the

observation

that

the

ith

word

is

the

mth

word

in

the

lexicon

and

z

i

x

i

represent

all

topic

and

author

assignments

not

including

the

ith

word

Furthermore

C

W

T

mj

is

the

number

of

times

word

m

is

assigned

to

topic

j

not

including

the

current

instance

and

C

AT

kj

is

the

number

of

times

author

k

is

assigned

to

topic

j

not

including

the

current

instance

and

V

is

the

size

of

the

lexicon

During

parameter

estimation

the

algorithm

only

needs

to

keep

track

of

a

V

T

-LRB-

word

by

topic

-RRB-

count

matrix

and

a

K

T

-LRB-

author

by

topic

-RRB-

count

matrix

both

of

which

can

be

represented

efficiently

in

sparse

format

From

these

count

matrices

we

can

easily

estimate

the

topic-word

distributions

and

author-topic

distributions

by

mj

C

W

T

mj

m

C

W

T

m

j

V

-LRB-

-RRB-

kj

C

AT

kj

j

C

AT

kj

T

-LRB-

-RRB-

where

mj

is

the

probability

of

using

word

m

in

topic

j

and

kj

is

the

probability

of

using

topic

j

by

author

k

These

values

correspond

to

the

predictive

distributions

over

new

words

w

and

new

topics

z

conditioned

on

w

and

z

We

start

the

algorithm

by

assigning

words

to

random

topics

and

authors

-LRB-

from

the

set

of

authors

on

the

document

-RRB-

Each

Gibbs

sample

then

constitutes

applying

Equation

-LRB-

-RRB-

to

every

word

token

in

the

document

collection

This

sampling

process

is

repeated

for

I

iterations

In

this

paper

we

primarily

focus

on

results

based

on

a

single

sample

so

that

specific

topics

can

be

identified

and

interpreted

in

tasks

involving

prediction

of

words

and

authors

one

can

average

over

topics

and

use

multiple

samples

when

doing

so

-LRB-

Rosen-Zvi

et

al.

-RRB-

Research

Track

Paper

WORD

PROB

WORD

PROB

WORD

PROB

WORD

PROB

PATTERNS

USER

MAGNETIC

METHODS

PATTERN

INTERFACE

STARS

METHOD

MATCHING

USERS

SOLAR

TECHNIQUES

MATCH

INTERFACES

EMISSION

DEVELOPED

TEXT

SYSTEM

MASS

APPLIED

PRESENT

INTERACTION

OBSERVATIONS

BASED

MATCHES

INTERACTIVE

STAR

APPROACHES

PAPER

USABILITY

RAY

COMPARE

SHOW

GRAPHICAL

GALAXIES

PRACTICAL

APPROACH

PROTOTYPE

OBSERVED

STANDARD

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

WORD

PROB

WORD

PROB

WORD

PROB

WORD

PROB

DATA

PROBABILISTIC

RETRIEVAL

QUERY

MINING

BAYESIAN

INFORMATION

QUERIES

DISCOVERY

PROBABILITY

TEXT

DATABASE

ATTRIBUTES

MODEL

DOCUMENTS

DATABASES

ASSOCIATION

MODELS

INDEXING

DATA

LARGE

PROBABILITIES

DOCUMENT

RELATIONAL

DATABASES

INFERENCE

QUERY

JOIN

KNOWLEDGE

CONDITIONAL

CONTENT

PROCESSING

PATTERNS

PRIOR

SEARCH

SOURCES

ITEMS

POSTERIOR

RELEVANCE

OPTIMIZATION

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

TOPIC

TOPIC

TOPIC

TOPIC

TOPIC

TOPIC

TOPIC

TOPIC

Figure

Eight

example

topics

extracted

from

the

CiteSeer

database

Each

is

illustrated

with

the

most

likely

words

and

authors

with

corresponding

probabilities

WORD

PROB

WORD

PROB

WORD

PROB

WORD

PROB

DATA

PROBABILISTIC

RETRIEVAL

QUERY

MINING

BAYESIAN

INFORMATION

QUERIES

DISCOVERY

PROBABILITY

INDEX

JOIN

ATTRIBUTES

MODEL

INDEXING

DATA

ASSOCIATION

PROBABILITIES

QUERY

OPTIMIZATION

RULES

INFERENCE

CONTENT

PROCESSING

PATTERNS

MODELS

BASED

RELATIONAL

LARGE

CONDITIONAL

SEARCH

DATABASE

ATTRIBUTE

DISTRIBUTION

RELEVANCE

AGGREGATION

DATABASES

PRIOR

SIMILARITY

RESULT

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

AUTHOR

PROB

TOPIC

TOPIC

TOPIC

TOPIC

Figure

The

four

most

similar

topics

to

the

topics

in

the

bottom

row

of

Figure

obtained

from

a

different

Markov

chain

run

Our

collection

of

CiteSeer

abstracts

contains

D

abstracts

with

K

authors

We

preprocessed

the

text

by

removing

all

punctuation

and

common

stop

words

This

led

to

a

vocabulary

size

of

V

and

a

total

of

word

tokens

There

is

inevitably

some

noise

in

data

of

this

form

given

that

many

of

the

fields

-LRB-

paper

title

author

names

year

abstract

-RRB-

were

extracted

automatically

by

CiteSeer

from

PDF

or

postscript

or

other

document

formats

We

chose

the

simple

convention

of

identifying

authors

by

their

first

initial

and

second

name

e.g.

A

Einstein

given

that

multiple

first

initials

or

fully

spelled

first

names

were

only

available

for

a

relatively

small

fraction

of

papers

This

means

of

course

that

for

some

very

common

names

-LRB-

e.g.

J

Wang

or

J

Smith

-RRB-

there

will

be

multiple

actual

individuals

represented

by

a

single

name

in

the

model

This

is

a

known

limitation

of

working

with

this

type

of

data

-LRB-

e.g.

see

Newman

-LRB-

-RRB-

for

further

discussion

-RRB-

There

are

algorithmic

techniques

that

could

be

used

to

automatically

resolve

these

identity

problems-however

in

this

paper

we

do

n't

pursue

these

options

and

instead

for

simplicity

work

with

the

first-initial/last-name

representation

of

individual

authors

In

our

simulations

the

number

of

topics

T

was

fixed

at

and

the

smoothing

parameters

and

-LRB-

Figure

-RRB-

were

set

at

and

respectively

We

ran

independent

Gibbs

sampling

chains

for

iterations

each

On

a

GHz

PC

workstation

each

iteration

took

seconds

leading

to

a

total

run

time

on

the

order

of

several

days

per

chain

Author-Topic

and

Topic-Word

Models

for

the

CiteSeer

Database

We

now

discuss

the

author-topic

and

topic-word

distributions

learned

from

the

CiteSeer

data

Figure

illustrates

eight

different

topics

-LRB-

out

of

-RRB-

obtained

at

the

iteration

of

a

particular

Gibbs

sampler

run

Each

table

in

Figure

shows

the

words

that

are

most

likely

to

be

produced

if

that

topic

is

activated

and

the

authors

who

are

most

likely

to

have

produced

a

word

if

it

is

known

to

have

come

from

that

topic

The

words

associated

with

each

topic

are

quite

intuitive

and

indeed

quite

precise

in

the

sense

of

conveying

a

semantic

summary

of

a

particular

field

of

research

The

authors

associated

with

each

topic

are

also

quite

representative

note

that

the

top

authors

associated

with

a

topic

by

the

model

are

not

necessarily

the

most

well-known

authors

in

that

area

but

rather

are

the

authors

who

tend

to

produce

the

most

words

for

that

topic

-LRB-

in

the

CiteSeer

abstracts

-RRB-

The

first

topics

at

the

top

of

Figure

topics

and

show

examples

of

quite

specific

and

precise

topics

on

string

matching

human-computer

interaction

and

astronomy

respectively

The

bottom

four

topics

-LRB-

and

-RRB-

are

examples

of

topics

with

direct

relevance

to

data

mining

namely

data

mining

itself

probabilistic

learning

information

retrieval

and

database

querying

and

indexing

The

model

includes

several

other

topics

related

to

data

mining

such

as

predictive

modeling

and

neural

networks

as

well

as

topics

that

span

the

full

range

of

research

areas

encompassed

by

documents

in

CiteSeer

The

full

list

is

available

at

Topic

-LRB-

top

right

Figure

-RRB-

provides

an

example

of

a

Research

Track

Paper

topic

that

is

not

directly

related

to

a

specific

research

area

A

fraction

of

topics

perhaps

to

are

devoted

to

non-research

specific

topics

the

glue

that

makes

up

our

research

papers

including

general

terminology

for

describing

methods

and

experiments

funding

acknowledgments

and

parts

of

addresses

-LRB-

which

inadvertently

crept

in

to

the

abstracts

-RRB-

and

so

forth

We

found

that

the

topics

obtained

from

different

Gibbs

sampling

runs

were

quite

stable

For

example

Figure

shows

the

most

similar

topics

to

the

topics

in

the

bottom

row

of

Figure

but

from

a

different

run

There

is

some

variability

in

terms

of

ranking

of

specific

words

and

authors

for

each

topic

and

in

the

exact

values

of

the

associated

probabilities

but

overall

the

topics

match

very

closely

Of

the

original

abstracts

in

our

data

set

estimated

years

of

publication

were

provided

by

CiteSeer

for

of

these

abstracts

There

is

a

steady

-LRB-

and

well-known

-RRB-

increase

year

by

year

in

the

number

of

online

documents

through

the

's

From

through

however

the

number

of

documents

for

which

the

year

is

known

drops

off

sharply-the

years

and

in

particular

are

under-represented

in

this

set

This

is

due

to

fact

that

it

is

easier

for

CiteSeer

to

determine

the

date

of

publication

of

older

documents

e.g.

by

using

citations

to

these

documents

We

used

the

yearly

data

to

analyze

trends

in

topics

over

time

Using

the

same

topic

model

described

earlier

the

documents

were

partitioned

by

year

and

for

each

year

all

of

the

words

were

assigned

to

their

most

likely

topic

using

the

model

The

fraction

of

words

assigned

to

each

topic

for

a

given

year

was

then

calculated

for

each

of

the

topics

and

for

each

year

from

to

These

fractions

provide

interesting

and

useful

indicators

of

relative

topic

popularity

in

the

research

literature

in

recent

years

Figure

shows

the

results

of

plotting

several

different

topics

Each

topic

is

indicated

in

the

legend

by

the

five

most

probable

words

in

the

topic

The

top

left

plot

shows

a

steady

increase

-LRB-

roughly

three-fold

-RRB-

in

machine

learning

and

data

mining

topics

The

top

right

plot

shows

a

tale

of

two

topics

an

increase

in

information-retrieval

coupled

to

an

apparent

decrease

in

natural

language

processing

On

the

second

row

on

the

left

we

see

a

steady

decrease

in

two

classical

computer

science

topics

operating

systems

and

programming

languages

On

the

right

however

we

see

the

reverse

behavior

namely

a

corresponding

substantial

growth

in

Web-related

topics

In

the

third

row

the

left

plot

illustrates

trends

within

database

research

a

decrease

in

the

transaction

and

concurrency-related

topic

query-related

research

holding

steady

over

time

and

a

slow

but

steady

increase

in

integration-related

database

research

The

plot

on

the

right

in

the

third

row

illustrates

the

changing

fortunes

of

security-related

research

a

decline

in

the

early

's

but

then

a

seemingly

dramatic

upward

trend

starting

around

The

lower

left

plot

on

the

bottom

row

illustrates

the

somewhat

noisy

trends

of

three

topics

that

were

hot

in

the

's

neural

networks

exhibits

a

steady

decline

since

the

early

's

-LRB-

as

machine

learning

has

moved

on

to

areas

such

as

support

vector

machines

-RRB-

genetic

algorithms

appears

to

be

relatively

stable

and

wavelets

may

have

peaked

in

the

time

period

Finally

as

with

any

large

data

set

there

are

always

some

surprises

in

store

The

final

figure

on

the

bottom

right

shows

two

somewhat

unexpected

topics

The

first

topic

consists

entirely

of

French

words

-LRB-

in

fact

the

model

discovered

such

French

language

topics

-RRB-

The

apparent

peaking

of

French

words

in

the

is

likely

to

be

an

artifact

of

how

CiteSeer

preprocesses

data

rather

than

any

indication

of

French

research

productivity

The

lower

curve

corresponds

to

a

topic

consisting

of

largely

Greek

letters

presumably

from

more

theoretically

oriented

papers

fans

of

theory

may

be

somewhat

dismayed

to

see

that

there

is

an

apparent

steady

decline

in

the

relative

frequency

of

Greek

letters

in

abstracts

since

the

The

time-trend

results

above

should

be

interpreted

with

some

caution

As

mentioned

earlier

the

data

for

and

are

relatively

sparse

compared

to

earlier

years

In

addition

the

numbers

are

based

on

a

rather

skewed

sample

-LRB-

online

documents

obtained

by

the

CiteSeer

system

for

which

years

are

known

-RRB-

Furthermore

the

fractions

per

year

only

indicate

the

relative

number

of

words

assigned

to

a

topic

by

the

model

and

make

no

direct

assessment

of

the

quality

or

importance

of

a

particular

sub-area

of

computer

science

Nonetheless

despite

these

caveats

the

results

are

quite

informative

and

indicate

substantial

shifts

in

research

topics

within

the

field

of

computer

science

In

terms

of

related

work

Popescul

et

al.

-LRB-

-RRB-

investi-gated

time

trends

in

CiteSeer

documents

using

a

document

clustering

approach

documents

were

clustered

into

clusters

based

on

co-citation

information

while

the

text

information

in

the

documents

was

not

used

Our

author-topic

model

uses

the

opposite

approach

In

effect

we

use

the

text

information

directly

to

discover

topics

and

do

not

explic-itly

model

the

author

network

-LRB-

although

implicitly

the

co-author

connections

are

used

by

the

model

-RRB-

A

direct

quantitative

comparison

is

difficult

but

we

can

say

that

our

model

with

topics

appears

to

produce

much

more

noticeable

and

precise

time-trends

than

the

model

Topics

and

Authors

for

New

Documents

In

many

applications

we

would

like

to

quickly

assess

the

topic

and

author

assignments

for

new

documents

not

contained

in

our

subset

of

the

CiteSeer

collection

Because

our

Monte

Carlo

algorithm

requires

significant

processing

time

for

documents

it

would

be

computationally

inefficient

to

rerun

the

algorithm

for

every

new

document

added

to

the

collection

-LRB-

even

though

from

a

Bayesian

inference

viewpoint

this

is

the

optimal

approach

-RRB-

Our

strategy

instead

is

to

apply

an

efficient

Monte

Carlo

algorithm

that

runs

only

on

the

word

tokens

in

the

new

document

leading

quickly

to

likely

assignments

of

words

to

authors

and

topics

We

start

by

assigning

words

randomly

to

co-authors

and

topics

We

then

sample

new

assignments

of

words

to

topics

and

authors

by

applying

Equation

only

to

the

word

tokens

in

the

new

document

each

time

temporarily

updating

the

count

matrices

C

W

T

and

C

AT

The

resulting

assignments

of

words

to

authors

and

topics

can

be

saved

after

a

few

iterations

-LRB-

iterations

in

our

simulations

-RRB-

Figure

shows

an

example

of

this

type

of

inference

Abstracts

from

two

authors

B

Scholkopf

and

A

Darwiche

were

combined

together

into

pseudo-abstract

and

the

docu-Research

Track

Paper

x

Year

Fraction

of

Words

Assigned

to

Topic

regression-variance-estimator

estimators-bias

classification-training-classifier

classifiers-generalization

data-mining-attributes-discovery

association

x

Year

Fraction

of

Words

Assigned

to

Topic

language-semantic-natural

linguistic-grammar

retrieval-text-documents

information-document

x

Year

Fraction

of

Words

Assigned

to

Topic

programming-language-concurrent

languages-implementation

system-operating-file

systems-kernel

Year

Fraction

of

Words

Assigned

to

Topic

web-user-world-wide-users

mobile-wireless-devices

mobility-ad

multicast-multimedia-media

delivery-applications

x

Year

Fraction

of

Words

Assigned

to

Topic

query-queries-index-data-join

transaction-transactions

concurrency-copy-copies

integration-view-views-data

incremental

x

Year

Fraction

of

Words

Assigned

to

Topic

security-secure-access-key-authentication

key-attack-encryption-hash-keys

x

Year

Fraction

of

Words

Assigned

to

Topic

neural-networks-network-training-learning

wavelet-operator-operators-basis

coefficients

genetic-evolutionary-evolution-population-ga

Year

Fraction

of

Words

Assigned

to

Topic

la-les-une-nous-est

gamma-delta-ff-omega-oe

Figure

Topic

trends

for

research

topics

in

computer

science

Research

Track

Paper

-LSB-

-LRB-

-RRB-

-RSB-

-LSB-

-LRB-

-RRB-

-RSB-

A

method

is

described

which

like

the

kernel

trick

in

support

vector

machines

SVMs

lets

us

generalize

distance

based

algorithms

to

operate

in

feature

spaces

usually

nonlinearly

related

to

the

input

space

This

is

done

by

identifying

a

class

of

kernels

which

can

be

represented

as

norm

based

distances

in

Hilbert

spaces

It

turns

out

that

common

kernel

algorithms

such

as

SVMs

and

kernel

PCA

are

actually

really

distance

based

algorithms

and

can

be

run

with

that

class

of

kernels

too

As

well

as

providing

a

useful

new

insight

into

how

these

algorithms

work

the

present

work

can

form

the

basis

for

conceiving

new

algorithms

This

paper

presents

a

comprehensive

approach

for

model

based

diagnosis

which

includes

proposals

for

characterizing

and

computing

preferred

diagnoses

assuming

that

the

system

description

is

augmented

with

a

system

structure

a

directed

graph

explicating

the

interconnections

between

system

components

Specifically

we

first

introduce

the

notion

of

a

consequence

which

is

a

syntactically

unconstrained

propositional

sentence

that

characterizes

all

consistency

based

diagnoses

and

show

that

standard

characterizations

of

diagnoses

such

as

minimal

conflicts

correspond

to

syntactic

variations

on

a

consequence

Second

we

propose

a

new

syntactic

variation

on

the

consequence

known

as

negation

normal

form

NNF

and

discuss

its

merits

compared

to

standard

variations

Third

we

introduce

a

basic

algorithm

for

computing

consequences

in

NNF

given

a

structured

system

description

We

show

that

if

the

system

structure

does

not

contain

cycles

then

there

is

always

a

linear

size

consequence

in

NNF

which

can

be

computed

in

linear

time

For

arbitrary

system

structures

we

show

a

precise

connection

between

the

complexity

of

computing

consequences

and

the

topology

of

the

underlying

system

structure

Finally

we

present

an

algorithm

that

enumerates

the

preferred

diagnoses

characterized

by

a

consequence

The

algorithm

is

shown

to

take

linear

time

in

the

size

of

the

consequence

if

the

preference

criterion

satisfies

some

general

conditions

Figure

Automated

labeling

of

a

pseudo-abstract

from

two

authors

by

the

model

ment

treated

as

if

they

had

both

written

it

These

two

authors

work

in

relatively

different

but

not

entirely

unrelated

sub-areas

of

computer

science

Scholkopf

in

machine

learning

and

Darwiche

in

probabilistic

reasoning

The

document

is

then

parsed

by

the

model

i.e.

words

are

assigned

to

these

authors

We

would

hope

that

the

author-topic

model

conditioned

now

on

these

two

authors

can

separate

the

combined

abstract

into

its

component

parts

Figure

shows

the

results

after

the

model

has

classified

each

word

according

to

the

most

likely

author

Note

that

the

model

only

sees

a

bag

of

words

and

is

not

aware

of

the

word

order

that

we

see

in

the

figure

For

readers

viewing

this

in

color

the

more

red

a

word

is

the

more

likely

it

is

to

have

been

generated

-LRB-

according

to

the

model

-RRB-

by

Scholkopf

-LRB-

and

blue

for

Darwiche

-RRB-

For

readers

viewing

the

figure

in

black

and

white

the

superscript

indicates

words

classified

by

the

model

for

Scholkopf

and

superscript

for

Darwiche

The

results

show

that

all

of

the

significant

content

words

-LRB-

such

as

kernel

support

vector

diagnoses

directed

graph

-RRB-

are

classified

correctly

As

we

might

expect

most

of

the

er-rors

are

words

-LRB-

such

as

based

or

criterion

-RRB-

that

are

not

specific

to

either

authors

area

of

research

Were

we

to

use

word

order

in

the

classification

and

classify

-LRB-

for

example

-RRB-

whole

sentences

the

accuracy

would

increase

further

As

it

is

the

model

correctly

classifies

of

Scholkopf

's

words

and

of

Darwiche

's

Detecting

the

Most

Surprising

and

Least

Surprising

Papers

for

an

Author

In

Tables

through

we

used

the

model

to

score

papers

attributed

to

three

well-known

researchers

in

computer

science

-LRB-

Christos

Faloutsos

Michael

Jordan

and

Tom

Mitchell

-RRB-

For

each

document

for

each

of

these

authors

we

calculate

a

perplexity

score

Perplexity

is

widely

used

in

language

modeling

to

assess

the

predictive

power

of

a

model

It

is

a

measure

of

how

surprising

the

words

are

from

the

model

's

perspective

loosely

equivalent

to

the

effective

branching

factor

Formally

the

perplexity

score

of

a

new

unobserved

document

d

that

contains

a

set

of

words

W

d

and

conditioned

on

a

topic

model

for

a

specific

author

a

is

Perplexity

-LRB-

W

d

a

-RRB-

exp

log

p

-LRB-

W

d

a

-RRB-

W

d

where

p

-LRB-

W

d

a

-RRB-

is

the

probability

assigned

by

the

author

topic

model

to

the

words

W

d

conditioned

on

the

single

author

a

and

W

d

is

the

number

of

words

in

the

document

Even

if

the

document

was

written

by

multiple

authors

we

evaluate

the

perplexity

score

relative

to

a

single

author

in

order

to

judge

perplexity

relative

to

that

individual

Our

goal

here

is

not

to

evaluate

the

out-of-sample

predictive

power

of

the

model

but

to

explore

the

range

of

perplexity

scores

that

the

model

assigns

to

papers

from

specific

authors

Lower

scores

imply

that

the

words

w

are

less

surprising

to

the

model

-LRB-

lower

bounded

by

zero

-RRB-

In

particular

we

are

interested

in

the

abstracts

that

the

model

considers

most

surprising

-LRB-

highest

perplexity

-RRB-

and

least

surprising

-LRB-

lowest

perplexity

-RRB-

in

each

table

we

list

the

abstracts

with

the

highest

perplexity

scores

the

median

perplexity

and

the

abstracts

with

the

lowest

perplexity

scores

Table

for

Christos

Faloutsos

shows

that

the

two

papers

with

the

highest

perplexities

have

significantly

higher

perplexity

scores

than

the

median

and

the

two

lowest

perplexity

papers

The

high

perplexity

papers

are

related

to

query

by

example

and

the

QBIC

image

database

system

while

the

low

perplexity

papers

are

on

high-dimensional

indexing

As

far

as

the

topic

model

for

Faloutsos

is

concerned

the

indexing

papers

are

much

more

typical

of

his

work

than

the

query

by

example

papers

Tables

and

provide

interesting

examples

in

that

the

most

perplexing

papers

-LRB-

from

the

model

's

viewpoint

-RRB-

for

each

author

are

papers

that

the

author

did

not

write

at

all

As

mentioned

earlier

by

combining

all

T

Mitchell

's

and

M

Jordan

's

together

the

data

set

may

contain

authors

who

are

different

from

Tom

Mitchell

at

CMU

and

Michael

Jordan

at

Berkeley

Thus

the

highest

perplexity

paper

for

T

Mitchell

is

in

fact

authored

by

a

Toby

Mitchell

and

is

on

the

topic

of

estimating

radiation

doses

-LRB-

quite

different

from

the

machine

learning

work

of

Tom

Mitchell

-RRB-

Similarly

for

Michael

Jordan

the

most

perplexing

paper

is

on

software

Research

Track

Paper

Table

Papers

ranked

by

perplexity

for

C.

Faloutsos

from

documents

Paper

Title

Perplexity

Score

MindReader

Querying

databases

through

multiple

examples

Efficient

and

effective

querying

by

image

content

MEDIAN

SCORE

Beyond

uniformity

and

independence

analysis

of

R-trees

using

the

concept

of

fractal

dimension

The

TV-tree

an

index

structure

for

high-dimensional

data

Table

Papers

ranked

by

perplexity

for

M.

Jordan

from

documents

Paper

Title

Perplexity

Score

Software

configuration

management

in

an

object

oriented

database

Are

arm

trajectories

planned

in

kinematic

or

dynamic

coordinates

An

adaptation

study

MEDIAN

SCORE

On

convergence

properties

of

the

EM

algorithm

for

Gaussian

mixtures

Supervised

learning

from

incomplete

data

via

an

EM

approach

Table

Papers

ranked

by

perplexity

for

T.

Mitchell

from

documents

Paper

Title

Perplexity

Score

A

method

for

estimating

occupational

radiation

dose

to

individuals

using

weekly

dosimetry

data

Text

classification

from

labeled

and

unlabeled

documents

using

EM

MEDIAN

SCORE

Learning

one

more

thing

Explanation

based

learning

for

mobile

robot

perception

configuration

management

and

was

written

by

Mick

Jordan

of

Sun

Microsystems

In

fact

of

the

most

perplexing

papers

for

M

Jordan

are

on

software

management

and

the

JAVA

programming

language

all

written

by

Mick

Jordan

However

the

most

perplexing

paper

was

in

fact

coauthored

by

Michael

Jordan

but

in

the

area

of

modeling

of

motor

planning

which

is

a

far

less

common

topic

compared

to

the

machine

learning

papers

that

Jordan

typically

writes

We

have

built

a

JAVA-based

query

interface

tool

that

supports

interactive

querying

of

the

model

The

tool

allows

a

user

to

query

about

authors

topics

documents

or

words

For

example

given

a

query

on

a

particular

author

the

tool

retrieves

and

displays

the

most

likely

topics

and

their

probabilities

for

that

author

the

most

probable

words

for

each

topic

and

the

document

titles

in

the

database

for

that

author

Figure

-LRB-

a

-RRB-

-LRB-

top

panel

-RRB-

shows

the

result

of

querying

on

Pazzani

M

and

the

resulting

topic

distribution

-LRB-

highly-ranked

topics

include

machine

learning

classification

rule-based

systems

data

mining

and

information

retrieval

-RRB-

Mouse-clicking

on

one

of

the

topics

-LRB-

e.g.

the

data

mining

topic

as

shown

in

the

figure

-RRB-

produces

the

screen

display

to

the

left

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

The

most

likely

words

for

this

topic

and

the

most

likely

authors

given

a

word

from

this

topic

are

then

displayed

We

have

found

this

to

be

a

useful

technique

for

interactively

exploring

topics

and

authors

e.g.

which

authors

are

active

in

a

particular

research

area

Similarly

one

can

click

on

a

particular

paper

-LRB-

e.g.

the

paper

A

Learning

Agent

for

Wireless

News

Access

as

shown

in

the

lower

screenshot

-LRB-

Figure

-LRB-

c

-RRB-

-RRB-

and

the

display

in

the

panel

to

the

right

is

then

produced

This

display

shows

the

words

in

the

documents

and

their

counts

the

probability

distribution

over

topics

for

the

paper

given

the

word

counts

A

prototype

online

version

of

the

tool

can

be

accessed

at

-LRB-

ranked

by

highest

probability

first

-RRB-

and

a

probability

distribution

over

authors

based

on

the

proportion

of

words

assigned

by

the

model

to

each

topic

and

author

respectively

The

system

is

implemented

using

a

combination

of

a

relational

database

and

real-time

Bayesian

estimation

-LRB-

a

relatively

rare

combination

of

these

technologies

for

a

real-time

query-answering

system

as

far

as

we

are

aware

-RRB-

We

use

a

database

to

store

and

index

both

-LRB-

a

-RRB-

the

sparse

author-topic

and

topic-word

count

matrices

that

are

learned

by

our

algorithm

from

the

training

data

and

-LRB-

b

-RRB-

various

tables

describing

the

data

such

as

document-word

document-author

and

document-title

tables

For

a

large

document

set

such

as

CiteSeer

-LRB-

and

with

topics

-RRB-

these

tables

can

run

into

the

hundred

's

of

megabytes

of

memory

thus

we

do

not

load

them

into

main

memory

automatically

but

instead

issue

SQL

commands

to

retrieve

the

relevant

records

in

real-time

For

most

of

the

queries

we

have

implemented

to

date

the

queries

can

be

answered

by

simple

table

lookup

followed

by

appropriate

normalization

-LRB-

if

needed

-RRB-

of

the

stored

counts

to

generate

conditional

probabilities

For

example

displaying

the

topic

distribution

for

a

specific

author

is

simply

a

matter

of

retrieving

the

appropriate

record

However

when

a

document

is

the

basis

of

a

query

-LRB-

e.g.

as

in

the

lower

screenshot

Figure

-LRB-

c

-RRB-

-RRB-

we

must

compute

in

real-time

the

conditional

distribution

of

the

fraction

of

words

assigned

to

each

topic

and

author

a

calculation

that

can

not

be

computed

in

closed

form

This

requires

retrieving

all

the

relevant

word-topic

counts

for

the

words

in

the

document

via

SQL

then

executing

the

estimation

algorithm

outlined

in

Section

in

real-time

using

Gibbs

sampling

and

displaying

the

results

to

the

user

The

user

can

change

adjust

the

burn-in

time

the

number

of

samples

and

the

lag

time

in

the

sampling

algorithm

typically

we

have

found

that

as

few

as

Gibbs

samples

gives

quite

reasonable

results

-LRB-

and

takes

on

the

order

of

or

seconds

depending

on

the

machine

being

used

other

factors

-RRB-

Research

Track

Paper

-LRB-

b

-RRB-

-LRB-

a

-RRB-

-LRB-

c

-RRB-

Figure

Examples

of

screenshots

from

the

interactive

query

browser

for

the

author-topic

model

with

-LRB-

a

-RRB-

querying

on

author

Pazzani

M

-LRB-

b

-RRB-

querying

on

a

topic

-LRB-

data

mining

-RRB-

relevant

to

that

author

and

-LRB-

c

-RRB-

querying

on

a

particular

document

written

by

the

author

Research

Track

Paper

We

have

introduced

a

probabilistic

algorithm

that

can

that

can

automatically

extract

information

about

authors

topics

and

documents

from

large

text

corpora

The

method

uses

a

generative

probabilistic

model

that

links

authors

to

observed

words

in

documents

via

latent

topics

We

demon-strated

that

Bayesian

estimation

can

be

used

to

learn

such

author-topic

models

from

very

large

text

corpora

using

CiteSeer

abstracts

as

a

working

example

The

resulting

CiteSeer

author-topic

model

was

shown

to

extract

substantial

novel

hidden

information

from

the

set

of

abstracts

including

topic

time-trends

author-topic

relations

unusual

papers

for

specific

authors

and

so

forth

Other

potential

applications

not

discussed

here

include

recommending

potential

reviewers

for

a

paper

based

on

both

the

words

in

the

paper

and

the

names

of

the

authors

Even

though

the

underlying

probabilistic

model

is

quite

simple

and

ignores

several

aspects

of

real-world

document

generation

-LRB-

such

as

topic

correlation

author

interaction

and

so

forth

-RRB-

it

nonetheless

provides

a

useful

first

step

in

understanding

author-topic

structure

in

large

text

corpora

We

would

like

to

thank

Steve

Lawrence

C.

Lee

Giles

and

Isaac

Council

for

providing

the

CiteSeer

data

used

in

this

paper

We

also

thank

Momo

Alhazzazi

Amnon

Meyers

and

Joshua

O'Madadhain

for

assistance

in

software

development

and

data

preprocessing

The

research

in

this

paper

was

supported

in

part

by

the

National

Science

Foundation

under

Grant

via

the

Knowledge

Discovery

and

Dissemination

-LRB-

KD-D

-RRB-

program

Blei

D.

M.

Ng

A.

Y.

and

Jordan

M.

I.

-LRB-

-RRB-

Latent

Dirichlet

allocation

Journal

of

Machine

Learning

Research

pp.

Buntine

W.L.

-LRB-

-RRB-

Operations

for

learning

with

graphical

models

Journal

of

Artificial

Intelligence

Research

pp.

Cutting

D.

Karger

D.

R.

Pederson

J.

and

Tukey

J.

W.

-LRB-

-RRB-

Scatter/Gather

a

cluster-based

approach

to

browsing

large

document

collections

in

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

pp.

Deerwester

S.

C.

Dumais

S.

T.

Landauer

T.

K.

Furnas

G.

W.

and

Harshman

R.

A.

-LRB-

-RRB-

Indexing

by

latent

semantic

analysis

Journal

of

the

American

Society

of

Information

Science

-LRB-

-RRB-

pp.

Diederich

J.

Kindermann

J.

Leopold

E.

and

Paass

G.

-LRB-

-RRB-

Authorship

attribution

with

support

vector

machines

Applied

Intelligence

-LRB-

-RRB-

Erten

C.

Harding

P.

J.

Kobourov

S.

G.

Wampler

K.

and

Yee

G.

-LRB-

-RRB-

Exploring

the

computing

literature

using

temporal

graph

visualization

Technical

Report

Department

of

Computer

Science

University

of

Arizona

Gray

A.

Sallis

P.

MacDonell

S.

-LRB-

-RRB-

Software

forensics

Extending

authorship

analysis

techniques

to

computer

programs

Proceedings

of

the

Biannual

Conference

of

the

International

Association

of

Forensic

Linguists

-LRB-

IAFL

-RRB-

Durham

NC

Griffiths

T.

L.

and

Steyvers

M.

-LRB-

-RRB-

Finding

scientific

topics

Proceedings

of

the

National

Academy

of

Sciences

-LRB-

suppl

-RRB-

Hofmann

T.

-LRB-

-RRB-

Probabilistic

latent

semantic

indexing

in

Proceedings

of

the

International

Conference

on

Research

and

Development

in

Information

Retrieval

-LRB-

SIGIR

-RRB-

Kautz

H.

Selman

B.

and

Shah

M.

-LRB-

-RRB-

Referral

Web

Combining

social

networks

and

collaborative

filtering

Communications

of

the

ACM

pp.

Lagus

K

Honkela

T.

Kaski

S.

and

Kohonen

T.

-LRB-

-RRB-

WEBSOM

for

textual

data

mining

Artificial

Intelligence

Review

-LRB-

-RRB-

pp.

Lawrence

S.

Giles

C.

L.

and

Bollacker

K.

-LRB-

-RRB-

Digital

libraries

and

autonomous

citation

indexing

IEEE

Computer

-LRB-

-RRB-

pp.

McCallum

A.

Nigam

K.

and

Ungar

L.

-LRB-

-RRB-

Efficient

clustering

of

high-dimensional

data

sets

with

application

to

reference

matching

in

Proceedings

of

the

Sixth

ACM

SIGKDD

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

Mosteller

F.

and

Wallace

D.

-LRB-

-RRB-

Applied

Bayesian

and

Classical

Inference

The

Case

of

the

Federalist

Papers

Springer-Verlag

Mutschke

P.

-LRB-

-RRB-

Mining

networks

and

central

entities

in

digital

libraries

a

graph

theoretic

approach

applied

to

co-author

networks

Intelligent

Data

Analysis

Lecture

Notes

in

Computer

Science

Springer

Verlag

pp.

Newman

M.

E.

J.

-LRB-

-RRB-

Scientific

collaboration

networks

I.

Network

construction

and

fundamental

results

Physical

Review

E

Popescul

A.

Flake

G.

W.

Lawrence

S.

Ungar

L.

H.

and

Giles

C.

L.

-LRB-

-RRB-

Clustering

and

identifying

temporal

trends

in

document

databases

IEEE

Advances

in

Digital

Libraries

ADL

pp.

Rosen-Zvi

M.

Griffiths

T.

Steyvers

M.

Smyth

P.

-LRB-

-RRB-

The

author-topic

model

for

authors

and

documents

Proceedings

of

the

UAI

Conference

July

Thisted

B.

and

Efron

R.

-LRB-

-RRB-

Did

Shakespeare

write

a

newly

discovered

poem

Biometrika

pp.

White

S.

and

Smyth

P.

-LRB-

-RRB-

Algorithms

for

estimating

relative

importance

in

networks

in

Proceedings

of

the

Ninth

ACM

SIGKDD

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

Yang

Y.

-LRB-

-RRB-

An

evaluation

of

statistical

approaches

to

text

categorization

Information

Retrieval

pp.

Research

Track

Paper

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O