Machine

Learning

for

Information

Architecture

in

a

Large

Governmental

Website

This

paper

describes

ongoing

research

into

the

application

of

machine

learning

techniques

for

improving

access

to

governmental

information

in

complex

digital

libraries

Under

the

auspices

of

the

GovStat

Project

our

goal

is

to

identify

a

small

number

of

semantically

valid

concepts

that

adequately

spans

the

intellectual

domain

of

a

collection

The

goal

of

this

discovery

is

twofold

First

we

desire

a

practical

aid

for

information

architects

Second

automatically

derived

document-concept

relationships

are

a

necessary

precondition

for

real-world

deployment

of

many

dynamic

interfaces

The

current

study

compares

concept

learning

strategies

based

on

three

document

representations

keywords

titles

and

full-text

In

statistical

and

user-based

studies

human-created

keywords

provide

significant

improvements

in

concept

learning

over

both

title-only

and

full-text

representations

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Digital

Libraries

Systems

Issues

User

Issues

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval-Clustering

This

research

was

supported

by

NSF

EIA

grant

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

JCDL

June

Tucson

Arizona

USA

Copyright

ACM

Design

Experimentation

The

GovStat

Project

is

a

joint

effort

of

the

University

of

North

Carolina

Interaction

Design

Lab

and

the

University

of

Maryland

Human-Computer

Interaction

Lab

Citing

end-user

difficulty

in

finding

governmental

information

-LRB-

especially

statistical

data

-RRB-

online

the

project

seeks

to

create

an

integrated

model

of

user

access

to

US

government

statistical

information

that

is

rooted

in

realistic

data

models

and

innovative

user

interfaces

To

enable

such

models

and

interfaces

we

propose

a

data-driven

approach

based

on

data

mining

and

machine

learning

techniques

In

particular

our

work

analyzes

a

particular

digital

library

the

website

of

the

Bureau

of

Labor

Statistics

-LRB-

BLS

-RRB-

in

efforts

to

discover

a

small

number

of

linguistically

meaningful

concepts

or

bins

that

collectively

summarize

the

semantic

domain

of

the

site

The

project

goal

is

to

classify

the

site

's

web

content

according

to

these

inferred

concepts

as

an

initial

step

towards

data

filtering

via

active

user

interfaces

-LRB-

cf.

-LSB-

-RSB-

-RRB-

Many

digital

libraries

already

make

use

of

content

classification

both

explicitly

and

implicitly

they

divide

their

resources

manually

by

topical

relation

they

organize

content

into

hi-erarchically

oriented

file

systems

The

goal

of

the

present

research

is

to

develop

another

means

of

browsing

the

content

of

these

collections

By

analyzing

the

distribution

of

terms

across

documents

our

goal

is

to

supplement

the

agency

's

pre-existing

information

structures

Statistical

learning

technologies

are

appealing

in

this

context

insofar

as

they

stand

to

define

a

data-driven

as

opposed

to

an

agency-driven-navigational

structure

for

a

site

Our

approach

combines

supervised

and

unsupervised

learning

techniques

A

pure

document

clustering

-LSB-

-RSB-

approach

to

such

a

large

diverse

collection

as

BLS

led

to

poor

results

in

early

tests

-LSB-

-RSB-

But

strictly

supervised

techniques

-LSB-

-RSB-

are

inappropriate

too

Although

BLS

designers

have

defined

high-level

subject

headings

for

their

collections

as

we

discuss

in

Section

this

scheme

is

less

than

optimal

Thus

we

hope

to

learn

an

additional

set

of

concepts

by

letting

the

data

speak

for

themselves

The

remainder

of

this

paper

describes

the

details

of

our

concept

discovery

efforts

and

subsequent

evaluation

In

Section

we

describe

the

previously

existing

human-created

conceptual

structure

of

the

BLS

website

This

section

also

describes

evidence

that

this

structure

leaves

room

for

improvement

Next

-LRB-

Sections

-RRB-

we

turn

to

a

description

of

the

concepts

derived

via

content

clustering

under

three

document

representations

keyword

title

only

and

full-text

Section

describes

a

two-part

evaluation

of

the

derived

conceptual

structures

Finally

we

conclude

in

Section

by

outlining

upcoming

work

on

the

project

The

Bureau

of

Labor

Statistics

is

a

federal

government

agency

charged

with

compiling

and

publishing

statistics

pertaining

to

labor

and

production

in

the

US

and

abroad

Given

this

broad

mandate

the

BLS

publishes

a

wide

array

of

information

intended

for

diverse

audiences

The

agency

's

website

acts

as

a

clearinghouse

for

this

process

With

over

text/html

documents

-LRB-

and

many

more

documents

if

spreadsheets

and

typeset

reports

are

included

-RRB-

providing

access

to

the

collection

provides

a

steep

challenge

to

information

architects

The

Relation

Browser

The

starting

point

of

this

work

is

the

notion

that

access

to

information

in

the

BLS

website

could

be

improved

by

the

addition

of

a

dynamic

interface

such

as

the

relation

browser

described

by

Marchionini

and

Brunk

-LSB-

-RSB-

The

relation

browser

allows

users

to

traverse

complex

data

sets

by

iteratively

slicing

the

data

along

several

topics

In

Figure

we

see

a

prototype

instantiation

of

the

relation

browser

applied

to

the

FedStats

website

The

relation

browser

supports

information

seeking

by

allowing

users

to

form

queries

in

a

stepwise

fashion

slicing

and

re-slicing

the

data

as

their

interests

dictate

Its

motivation

is

in

keeping

with

Shneiderman

's

suggestion

that

queries

and

their

results

should

be

tightly

coupled

-LSB-

-RSB-

Thus

in

Figure

Relation

Browser

Prototype

ure

users

might

limit

their

search

set

to

those

documents

about

energy

Within

this

subset

of

the

collection

they

might

further

eliminate

documents

published

more

than

a

year

ago

Finally

they

might

request

to

see

only

documents

published

in

PDF

format

As

Marchionini

and

Brunk

discuss

capturing

the

publication

date

and

format

of

documents

is

trivial

But

successful

implementations

of

the

relation

browser

also

rely

on

topical

classification

This

presents

two

stumbling

blocks

for

system

designers

Information

architects

must

define

the

appropriate

set

of

topics

for

their

collection

Site

maintainers

must

classify

each

document

into

its

appropriate

categories

These

tasks

parallel

common

problems

in

the

metadata

community

defining

appropriate

elements

and

marking

up

documents

to

support

metadata-aware

information

access

Given

a

collection

of

over

documents

these

hurdles

are

especially

daunting

and

automatic

methods

of

approaching

them

are

highly

desirable

A

Pre-Existing

Structure

Prior

to

our

involvement

with

the

project

designers

at

BLS

created

a

shallow

classificatory

structure

for

the

most

important

documents

in

their

website

As

seen

in

Figure

the

BLS

home

page

organizes

top-level

documents

into

categories

These

include

topics

such

as

Employment

and

Unemployment

Productivity

and

Inflation

and

Spending

Figure

The

BLS

Home

Page

We

hoped

initially

that

these

pre-defined

categories

could

be

used

to

train

a

document

classifier

thus

automating

the

process

of

populating

the

relation

browser

altogether

However

this

approach

proved

unsatisfactory

In

personal

meetings

BLS

officials

voiced

dissatisfaction

with

the

existing

topics

Their

form

it

was

argued

owed

as

much

to

the

institutional

structure

of

BLS

as

it

did

to

the

inherent

topology

of

the

website

's

information

space

In

other

words

the

topics

reflected

official

divisions

rather

than

semantic

clusters

The

BLS

agents

suggested

that

re-designing

this

classification

structure

would

be

desirable

The

agents

misgivings

were

borne

out

in

subsequent

analysis

The

BLS

topics

comprise

a

shallow

classificatory

structure

each

of

the

top-level

categories

is

linked

to

a

small

number

of

related

pages

Thus

there

are

pages

associated

with

Inflation

Altogether

the

link

structure

of

this

classificatory

system

contains

documents

that

is

excluding

navigational

links

there

are

documents

linked

from

the

BLS

home

page

where

each

hyperlink

connects

a

document

to

a

topic

-LRB-

pages

can

be

linked

to

multiple

topics

-RRB-

Based

on

this

hyperlink

structure

we

defined

M

a

symmetric

matrix

where

m

ij

counts

the

number

of

topics

in

which

documents

i

and

j

are

both

classified

on

the

BLS

home

page

To

analyze

the

redundancy

inherent

in

the

pre-existing

structure

we

derived

the

principal

components

of

M

-LRB-

cf.

-LSB-

-RSB-

-RRB-

Figure

shows

the

resultant

scree

plot

Because

all

documents

belong

to

at

least

one

BLS

topic

A

scree

plot

shows

the

magnitude

of

the

k

th

eigenvalue

versus

its

rank

During

principal

component

analysis

scree

plots

visualize

the

amount

of

variance

captured

by

each

component

Eigenvalue

Rank

Eigenvlue

Magnitude

Figure

Scree

Plot

of

BLS

Categories

the

rank

of

M

is

guaranteed

to

be

less

than

or

equal

to

-LRB-

hence

eigenvalues

-RRB-

What

is

surprising

about

Figure

however

is

the

precipitous

decline

in

magnitude

among

the

first

four

eigenvalues

The

four

largest

eigenvlaues

account

for

of

the

total

variance

in

the

data

This

fact

suggests

a

high

degree

of

redundancy

among

the

topics

Topical

redundancy

is

not

in

itself

problematic

However

the

documents

in

this

very

shallow

classificatory

structure

are

almost

all

gateways

to

more

specific

information

Thus

the

listing

of

the

Producer

Price

Index

under

three

categories

could

be

confusing

to

the

site

's

users

In

light

of

this

potential

for

confusion

and

the

agency

's

own

request

for

redesign

we

undertook

the

task

of

topic

discovery

described

in

the

following

sections

To

aid

in

the

discovery

of

a

new

set

of

high-level

topics

for

the

BLS

website

we

turned

to

unsupervised

machine

learning

methods

In

efforts

to

let

the

data

speak

for

themselves

we

desired

a

means

of

concept

discovery

that

would

be

based

not

on

the

structure

of

the

agency

but

on

the

content

of

the

material

To

begin

this

process

we

crawled

the

BLS

website

downloading

all

documents

of

MIME

type

text/html

This

led

to

a

corpus

of

documents

Based

on

this

corpus

we

hoped

to

derive

k

topical

categories

such

that

each

document

d

i

is

assigned

to

one

or

more

classes

Document

clustering

-LRB-

cf.

-LSB-

-RSB-

-RRB-

provided

an

obvious

but

only

partial

solution

to

the

problem

of

automating

this

type

of

high-level

information

architecture

discovery

The

problems

with

standard

clustering

are

threefold

Mutually

exclusive

clusters

are

inappropriate

for

identifying

the

topical

content

of

documents

since

documents

may

be

about

many

subjects

Due

to

the

heterogeneity

of

the

data

housed

in

the

BLS

collection

-LRB-

tables

lists

surveys

etc.

-RRB-

many

doc-uments

terms

provide

noisy

topical

information

For

application

to

the

relation

browser

we

require

a

small

number

-LRB-

k

-RRB-

of

topics

Without

significant

data

reduction

term-based

clustering

tends

to

deliver

clusters

at

too

fine

a

level

of

granularity

In

light

of

these

problems

we

take

a

hybrid

approach

to

topic

discovery

First

we

limit

the

clustering

process

to

a

sample

of

the

entire

collection

described

in

Section

Working

on

a

focused

subset

of

the

data

helps

to

overcome

problems

two

and

three

listed

above

To

address

the

problem

of

mutual

exclusivity

we

combine

unsupervised

with

supervised

learning

methods

as

described

in

Section

To

derive

empirically

evidenced

topics

we

initially

turned

to

cluster

analysis

Let

A

be

the

n

p

data

matrix

with

n

observations

in

p

variables

Thus

a

ij

shows

the

measurement

for

the

i

th

observation

on

the

j

th

variable

As

described

in

-LSB-

-RSB-

the

goal

of

cluster

analysis

is

to

assign

each

of

the

n

observations

to

one

of

a

small

number

k

groups

each

of

which

is

characterized

by

high

intra-cluster

correlation

and

low

inter-cluster

correlation

Though

the

algorithms

for

accomplishing

such

an

arrangement

are

legion

our

analysis

focuses

on

k-means

clustering

during

which

each

observation

o

i

is

assigned

to

the

cluster

C

k

whose

centroid

is

closest

to

it

in

terms

of

Euclidean

distance

Readers

interested

in

the

details

of

the

algorithm

are

referred

to

-LSB-

-RSB-

for

a

thorough

treatment

of

the

subject

Clustering

by

k-means

is

well-studied

in

the

statistical

literature

and

has

shown

good

results

for

text

analysis

-LRB-

cf.

-LSB-

-RSB-

-RRB-

However

k-means

clustering

requires

that

the

researcher

specify

k

the

number

of

clusters

to

define

When

applying

k-means

to

our

document

collection

indicators

such

as

the

gap

statistic

-LSB-

-RSB-

and

an

analysis

of

the

mean-squared

distance

across

values

of

k

suggested

that

k

was

optimal

This

paramterization

led

to

semantically

intelligible

clusters

However

clusters

are

far

too

many

for

application

to

an

interface

such

as

the

relation

We

have

focused

on

k-means

as

opposed

to

other

clustering

algorithms

for

several

reasons

Chief

among

these

is

the

computational

efficiency

enjoyed

by

the

k-means

approach

Because

we

need

only

a

flat

clustering

there

is

little

to

be

gained

by

the

more

expensive

hierarchical

algorithms

In

future

work

we

will

turn

to

model-based

clustering

-LSB-

-RSB-

as

a

more

principled

method

of

selecting

the

number

of

clusters

and

of

representing

clusters

browser

Moreover

the

granularity

of

these

clusters

was

un-suitably

fine

For

instance

the

solution

derived

a

cluster

whose

most

highly

associated

words

-LRB-

in

terms

of

log-odds

ratio

-LSB-

-RSB-

-RRB-

were

drug

pharmacy

and

chemist

These

words

are

certainly

related

but

they

are

related

at

a

level

of

specificity

far

below

what

we

sought

To

remedy

the

high

dimensionality

of

the

data

we

resolved

to

limit

the

algorithm

to

a

subset

of

the

collection

In

consultation

with

employees

of

the

BLS

we

continued

our

analysis

on

documents

that

form

a

series

titled

From

the

Editor

's

Desk

These

are

brief

articles

written

by

BLS

employees

BLS

agents

suggested

that

we

focus

on

the

Editor

's

Desk

because

it

is

intended

to

span

the

intellectual

domain

of

the

agency

The

column

is

published

daily

and

each

entry

describes

an

important

current

issue

in

the

BLS

domain

The

Editor

's

Desk

column

has

been

written

daily

-LRB-

five

times

per

week

-RRB-

since

As

such

we

operated

on

a

set

of

N

documents

Limiting

attention

to

these

documents

not

only

reduced

the

dimensionality

of

the

problem

It

also

allowed

the

clustering

process

to

learn

on

a

relatively

clean

data

set

While

the

entire

BLS

collection

contains

a

great

deal

of

non-prose

text

-LRB-

i.e.

tables

lists

etc.

-RRB-

the

Editor

's

Desk

documents

are

all

written

in

clear

journalistic

prose

Each

document

is

highly

topical

further

aiding

the

discovery

of

term-topic

relations

Finally

the

Editor

's

Desk

column

provided

an

ideal

learning

environment

because

it

is

well-supplied

with

topical

metadata

Each

of

the

documents

contains

a

list

of

one

or

more

keywords

Additionally

a

subset

of

the

documents

-LRB-

-RRB-

contained

a

subject

heading

This

metadata

informed

our

learning

and

evaluation

as

described

in

Section

To

derive

suitably

general

topics

for

the

application

of

a

dynamic

interface

to

the

BLS

collection

we

combined

document

clustering

with

text

classification

techniques

Specif-ically

using

k-means

we

clustered

each

of

the

documents

into

one

of

k

clusters

with

the

number

of

clusters

chosen

by

analyzing

the

within-cluster

mean

squared

distance

at

different

values

of

k

-LRB-

see

Section

-RRB-

Constructing

mutually

exclusive

clusters

violates

our

assumption

that

documents

may

belong

to

multiple

classes

However

these

clusters

mark

only

the

first

step

in

a

two-phase

process

of

topic

identification

At

the

end

of

the

process

document-cluster

affinity

is

measured

by

a

real-valued

number

Once

the

Editor

's

Desk

documents

were

assigned

to

clusters

we

constructed

a

k-way

classifier

that

estimates

the

strength

of

evidence

that

a

new

document

d

i

is

a

member

of

class

C

k

We

tested

three

statistical

classification

techniques

probabilistic

Rocchio

-LRB-

prind

-RRB-

naive

Bayes

and

support

vector

machines

-LRB-

SVMs

-RRB-

All

were

implemented

using

McCallum

's

BOW

text

classification

library

-LSB-

-RSB-

Prind

is

a

probabilistic

version

of

the

Rocchio

classification

algorithm

-LSB-

-RSB-

Interested

readers

are

referred

to

Joachims

article

for

further

details

of

the

classification

method

Like

prind

naive

Bayes

attempts

to

classify

documents

into

the

most

probable

class

It

is

described

in

detail

in

-LSB-

-RSB-

Finally

support

vector

machines

were

thoroughly

explicated

by

Vapnik

-LSB-

-RSB-

and

applied

specifically

to

text

in

-LSB-

-RSB-

They

define

a

decision

boundary

by

finding

the

maximally

separating

hyperplane

in

a

high-dimensional

vector

space

in

which

document

classes

become

linearly

separable

Having

clustered

the

documents

and

trained

a

suitable

classifier

the

remaining

documents

in

the

collection

are

labeled

by

means

of

automatic

classification

That

is

for

each

document

d

i

we

derive

a

k-dimensional

vector

quantifying

the

association

between

d

i

and

each

class

C

C

k

Deriving

topic

scores

via

naive

Bayes

for

the

entire

collection

required

less

than

two

hours

of

CPU

time

The

output

of

this

process

is

a

score

for

every

document

in

the

collection

on

each

of

the

automatically

discovered

topics

These

scores

may

then

be

used

to

populate

a

relation

browser

interface

or

they

may

be

added

to

a

traditional

information

retrieval

system

To

use

these

weights

in

the

relation

browser

we

currently

assign

to

each

document

the

two

topics

on

which

it

scored

highest

In

future

work

we

will

adopt

a

more

rigorous

method

of

deriving

document-topic

weight

thresholds

Also

evaluation

of

the

utility

of

the

learned

topics

for

users

will

be

undertaken

Prior

to

implementing

a

relation

browser

interface

and

undertaking

the

attendant

user

studies

it

is

of

course

important

to

evaluate

the

quality

of

the

inferred

concepts

and

the

ability

of

the

automatic

classifier

to

assign

documents

to

the

appropriate

subjects

To

evaluate

the

success

of

the

two-stage

approach

described

in

Section

we

undertook

two

experiments

During

the

first

experiment

we

compared

three

methods

of

document

representation

for

the

clustering

task

The

goal

here

was

to

compare

the

quality

of

document

clusters

derived

by

analysis

of

full-text

documents

documents

represented

only

by

their

titles

and

documents

represented

by

human-created

keyword

metadata

During

the

second

experiment

we

analyzed

the

ability

of

the

statistical

classifiers

to

discern

the

subject

matter

of

documents

from

portions

of

the

database

in

addition

to

the

Editor

's

Desk

Comparing

Document

Representations

Documents

from

The

Editor

's

Desk

column

came

supplied

with

human-generated

keyword

metadata

Additionally

The

titles

of

the

Editor

's

Desk

documents

tend

to

be

germane

to

the

topic

of

their

respective

articles

With

such

an

array

of

distilled

evidence

of

each

document

's

subject

matter

we

undertook

a

comparison

of

document

representations

for

topic

discovery

by

clustering

We

hypothesized

that

keyword-based

clustering

would

provide

a

useful

model

But

we

hoped

to

see

whether

comparable

performance

could

be

attained

by

methods

that

did

not

require

extensive

human

indexing

such

as

the

title-only

or

full-text

representations

To

test

this

hypothesis

we

defined

three

modes

of

document

representation

full-text

title-only

and

keyword

only

we

generated

three

sets

of

topics

T

f

ull

T

title

and

T

kw

respectively

Topics

based

on

full-text

documents

were

derived

by

application

of

k-means

clustering

to

the

Editor

's

Desk

documents

where

each

document

was

represented

by

a

vector

These

dimensions

captured

the

TF.IDF

weights

-LSB-

-RSB-

of

each

term

t

i

in

document

d

j

for

all

terms

that

occurred

at

least

three

times

in

the

data

To

arrive

at

the

appropriate

number

of

clusters

for

these

data

we

inspected

the

within-cluster

mean-squared

distance

for

each

value

of

k

As

k

approached

the

reduction

in

error

with

the

addition

of

more

clusters

declined

notably

suggesting

that

k

would

yield

good

divisions

To

select

a

single

integer

value

we

calculated

which

value

of

k

led

to

the

least

variation

in

cluster

size

This

metric

stemmed

from

a

desire

to

suppress

the

common

result

where

one

large

cluster

emerges

from

the

k-means

algorithm

accompanied

by

several

accordingly

small

clusters

Without

reason

to

believe

that

any

single

topic

should

have

dramatically

high

prior

odds

of

document

membership

this

heuristic

led

to

k

f

ull

Clusters

based

on

document

titles

were

constructed

simi-larly

However

in

this

case

each

document

was

represented

in

the

vector

space

spanned

by

the

terms

that

occur

at

least

twice

in

document

titles

Using

the

same

method

of

minimizing

the

variance

in

cluster

membership

k

title

the

number

of

clusters

in

the

title-based

representation

was

also

set

to

The

dimensionality

of

the

keyword-based

clustering

was

very

similar

to

that

of

the

title-based

approach

There

were

keywords

in

the

data

all

of

which

were

retained

The

median

number

of

keywords

per

document

was

where

a

keyword

is

understood

to

be

either

a

single

word

or

a

multi-word

term

such

as

consumer

price

index

It

is

worth

noting

that

the

keywords

were

not

drawn

from

any

controlled

vocabulary

they

were

assigned

to

documents

by

publishers

at

the

BLS

Using

the

keywords

the

documents

were

clustered

into

classes

To

evaluate

the

clusters

derived

by

each

method

of

document

representation

we

used

the

subject

headings

that

were

included

with

of

the

Editor

's

Desk

documents

Each

of

these

documents

was

assigned

one

or

more

subject

headings

which

were

withheld

from

all

of

the

cluster

applications

Like

the

keywords

subject

headings

were

assigned

to

documents

by

BLS

publishers

Unlike

the

keywords

however

subject

headings

were

drawn

from

a

controlled

vocabulary

Our

analysis

began

with

the

assumption

that

documents

with

the

same

subject

headings

should

cluster

together

To

facilitate

this

analysis

we

took

a

conservative

approach

we

considered

multi-subject

classifications

to

be

unique

Thus

if

document

d

i

was

assigned

to

a

single

subject

prices

while

document

d

j

was

assigned

to

two

subjects

international

comparisons

prices

documents

d

i

and

d

j

are

not

considered

to

come

from

the

same

class

Table

shows

all

Editor

's

Desk

subject

headings

that

were

assigned

to

at

least

documents

As

noted

in

the

table

Table

Top

Editor

's

Desk

Subject

Headings

Subject

Count

prices

unemployment

occupational

safety

health

international

comparisons

prices

manufacturing

prices

employment

productivity

consumer

expenditures

earnings

wages

employment

unemployment

compensation

costs

earnings

wages

metro

areas

benefits

compensation

costs

earnings

wages

occupations

employment

occupations

benefits

earnings

wage

regions

work

stoppages

earnings

wages

industries

Total

Table

Contingecy

Table

for

Three

Document

Representations

Representation

Right

Wrong

Accuracy

Full-text

Title

Keyword

there

were

such

subject

headings

which

altogether

covered

-LRB-

-RRB-

of

the

documents

with

subjects

assigned

These

document-subject

pairings

formed

the

basis

of

our

analysis

Limiting

analysis

to

subjects

with

N

kept

the

resultant

tests

suitably

robust

The

clustering

derived

by

each

document

representation

was

tested

by

its

ability

to

collocate

documents

with

the

same

subjects

Thus

for

each

of

the

subject

headings

in

Table

S

i

we

calculated

the

proportion

of

documents

assigned

to

S

i

that

each

clustering

co-classified

Further

we

assumed

that

whichever

cluster

captured

the

majority

of

documents

for

a

given

class

constituted

the

right

answer

for

that

class

For

instance

There

were

documents

whose

subject

heading

was

prices

Taking

the

BLS

editors

classifications

as

ground

truth

all

of

these

documents

should

have

ended

up

in

the

same

cluster

Under

the

full-text

representation

of

these

documents

were

clustered

into

category

while

were

in

category

and

documents

were

in

category

Taking

the

majority

cluster

as

the

putative

right

home

for

these

documents

we

consider

the

accuracy

of

this

clustering

on

this

subject

to

be

Repeating

this

process

for

each

topic

across

all

three

representations

led

to

the

contingency

table

shown

in

Table

The

obvious

superiority

of

the

keyword-based

clustering

evidenced

by

Table

was

borne

out

by

a

test

on

the

accuracy

proportions

Comparing

the

proportion

right

and

Table

Keyword-Based

Clusters

benefits

costs

international

jobs

plans

compensation

import

employment

benefits

costs

prices

jobs

employees

benefits

petroleum

youth

occupations

prices

productivity

safety

workers

prices

productivity

safety

earnings

index

output

health

operators

inflation

nonfarm

occupational

spending

unemployment

expenditures

unemployment

consumer

mass

spending

jobless

wrong

achieved

by

keyword

and

title-based

clustering

led

to

p

Due

to

this

result

in

the

remainder

of

this

paper

we

focus

our

attention

on

the

clusters

derived

by

analysis

of

the

Editor

's

Desk

keywords

The

ten

keyword-based

clusters

are

shown

in

Table

represented

by

the

three

terms

most

highly

associated

with

each

cluster

in

terms

of

the

log-odds

ratio

Additionally

each

cluster

has

been

given

a

label

by

the

researchers

Evaluating

the

results

of

clustering

is

notoriously

difficult

In

order

to

lend

our

analysis

suitable

rigor

and

utility

we

made

several

simplifying

assumptions

Most

problematic

is

the

fact

that

we

have

assumed

that

each

document

belongs

in

only

a

single

category

This

assumption

is

certainly

false

However

by

taking

an

extremely

rigid

view

of

what

constitutes

a

subject

that

is

by

taking

a

fully

qualified

and

often

multipart

subject

heading

as

our

unit

of

analysis

we

mitigate

this

problem

Analogically

this

is

akin

to

considering

the

location

of

books

on

a

library

shelf

Although

a

given

book

may

cover

many

subjects

a

classification

system

should

be

able

to

collocate

books

that

are

extremely

similar

say

books

about

occupational

safety

and

health

The

most

serious

liability

with

this

evaluation

then

is

the

fact

that

we

have

compressed

multiple

subject

headings

say

prices

international

into

single

subjects

This

flattening

obscures

the

multivalence

of

documents

We

turn

to

a

more

realistic

assessment

of

document-class

relations

in

Section

Accuracy

of

the

Document

Classifiers

Although

the

keyword-based

clusters

appear

to

classify

the

Editor

's

Desk

documents

very

well

their

discovery

only

solved

half

of

the

problem

required

for

the

successful

implementation

of

a

dynamic

user

interface

such

as

the

relation

browser

The

matter

of

roughly

fourteen

thousand

unclassified

documents

remained

to

be

addressed

To

solve

this

problem

we

trained

the

statistical

classifiers

described

above

in

Section

For

each

document

in

the

collection

d

i

these

classifiers

give

p

i

a

k-vector

of

probabilities

or

distances

-LRB-

depending

on

the

classification

method

used

-RRB-

where

p

ik

quantifies

the

strength

of

association

between

the

i

th

document

and

the

k

th

class

All

classifiers

were

trained

on

the

full

text

of

each

document

regardless

of

the

representation

used

to

discover

the

initial

clusters

The

different

training

sets

were

thus

constructed

simply

by

changing

the

Table

Cross

Validation

Results

for

Classifiers

Method

Av

Percent

Accuracy

SE

Prind

Naive

Bayes

SVM

class

variable

for

each

instance

-LRB-

document

-RRB-

to

reflect

its

assigned

cluster

under

a

given

model

To

test

the

ability

of

each

classifier

to

locate

documents

correctly

we

first

performed

a

cross

validation

on

the

Editor

's

Desk

documents

During

cross-validation

the

data

are

split

randomly

into

n

subsets

-LRB-

in

this

case

n

-RRB-

The

process

proceeds

by

iteratively

holding

out

each

of

the

n

subsets

as

a

test

collection

for

a

model

trained

on

the

remaining

n

subsets

Cross

validation

is

described

in

-LSB-

-RSB-

Using

this

methodology

we

compared

the

performance

of

the

three

classification

models

described

above

Table

gives

the

results

from

cross

validation

Although

naive

Bayes

is

not

significantly

more

accurate

for

these

data

than

the

SVM

classifier

we

limit

the

remainder

of

our

attention

to

analysis

of

its

performance

Our

selection

of

naive

Bayes

is

due

to

the

fact

that

it

appears

to

work

comparably

to

the

SVM

approach

for

these

data

while

being

much

simpler

both

in

theory

and

implementation

Because

we

have

only

documents

and

classes

the

number

of

training

documents

per

class

is

relatively

small

In

addition

to

models

fitted

to

the

Editor

's

Desk

data

then

we

constructed

a

fourth

model

supplementing

the

training

sets

of

each

class

by

querying

the

Google

search

engine

and

applying

naive

Bayes

to

the

augmented

test

set

For

each

class

we

created

a

query

by

submitting

the

three

terms

with

the

highest

log-odds

ratio

with

that

class

Further

each

query

was

limited

to

the

domain

For

each

class

we

retrieved

up

to

documents

from

Google

-LRB-

the

actual

number

varied

depending

on

the

size

of

the

result

set

returned

by

Google

-RRB-

This

led

to

a

training

set

of

documents

in

the

augmented

model

as

we

call

it

below

Cross

validation

suggested

that

the

augmented

model

decreased

classification

accuracy

-LRB-

accuracy

with

standard

error

-RRB-

As

we

discuss

below

however

augmenting

the

training

set

appeared

to

help

generalization

during

our

second

experiment

The

results

of

our

cross

validation

experiment

are

encouraging

However

the

success

of

our

classifiers

on

the

Editor

's

Desk

documents

that

informed

the

cross

validation

study

may

not

be

good

predictors

of

the

models

performance

on

the

remainder

to

the

BLS

website

To

test

the

generality

of

the

naive

Bayes

classifier

we

solicited

input

from

human

judges

who

were

familiar

with

the

BLS

website

The

sample

was

chosen

by

convenience

and

consisted

of

faculty

and

graduate

students

who

work

on

the

GovStat

project

However

of

the

reviewers

had

prior

knowledge

of

the

outcome

of

the

classification

before

their

participation

For

the

experiment

a

random

sample

of

documents

was

drawn

from

the

entire

BLS

collection

On

average

each

A

more

formal

treatment

of

the

combination

of

labeled

and

unlabeled

data

is

available

in

-LSB-

-RSB-

Table

Human-Model

Agreement

on

Sample

Docs

Human

Judge

st

Choice

Model

Model

st

Choice

Model

nd

Choice

N.

Bayes

-LRB-

aug.

-RRB-

N.

Bayes

Human

Judge

nd

Choice

Model

Model

st

Choice

Model

nd

Choice

N.

Bayes

-LRB-

aug.

-RRB-

N.

Bayes

viewer

classified

documents

placing

each

document

into

as

many

of

the

categories

shown

in

Table

as

he

or

she

saw

fit

Results

from

this

experiment

suggest

that

room

for

improvement

remains

with

respect

to

generalizing

to

the

whole

collection

from

the

class

models

fitted

to

the

Editor

's

Desk

documents

In

Table

we

see

for

each

classifier

the

number

of

documents

for

which

it

's

first

or

second

most

probable

class

was

voted

best

or

second

best

by

the

human

judges

In

the

context

of

this

experiment

we

consider

a

first

or

second-place

classification

by

the

machine

to

be

accurate

because

the

relation

browser

interface

operates

on

a

multi-way

classification

where

each

document

is

classified

into

multiple

categories

Thus

a

document

with

the

correct

class

as

its

second

choice

would

still

be

easily

available

to

a

user

Likewise

a

correct

classification

on

either

the

most

popular

or

second

most

popular

category

among

the

human

judges

is

considered

correct

in

cases

where

a

given

document

was

classified

into

multiple

classes

There

were

multi-class

documents

in

our

sample

as

seen

in

Figure

The

remaining

documents

were

assigned

to

or

classes

Under

this

rationale

The

augmented

naive

Bayes

classifier

correctly

grouped

documents

while

the

smaller

model

-LRB-

not

augmented

by

a

Google

search

-RRB-

correctly

classified

The

resultant

test

gave

p

suggesting

that

increasing

the

training

set

improved

the

ability

of

the

naive

Bayes

model

to

generalize

from

the

Editor

's

Desk

documents

to

the

collection

as

a

whole

However

the

improvement

afforded

by

the

augmented

model

comes

at

some

cost

In

particular

the

augmented

model

is

significantly

inferior

to

the

model

trained

solely

on

Editor

's

Desk

documents

if

we

concern

ourselves

only

with

documents

selected

by

the

majority

of

human

reviewers

i.e.

only

first-choice

classes

Limiting

the

right

answers

to

the

left

column

of

Table

gives

p

in

favor

of

the

non-augmented

model

For

the

purposes

of

applying

the

relation

browser

to

complex

digital

library

content

-LRB-

where

documents

will

be

classified

along

multiple

categories

-RRB-

the

augmented

model

is

preferable

But

this

is

not

necessarily

the

case

in

general

It

must

also

be

said

that

accuracy

under

a

fairly

liberal

test

condition

leaves

room

for

improvement

in

our

assignment

of

topics

to

categories

We

may

begin

to

understand

the

shortcomings

of

the

described

techniques

by

consulting

Figure

which

shows

the

distribution

of

categories

across

documents

given

by

humans

and

by

the

augmented

naive

Bayes

model

The

majority

of

reviewers

put

Number

of

Human-Assigned

Classes

Frequency

Figure

Number

of

Classes

Assigned

to

Documents

by

Judges

documents

into

only

three

categories

jobs

benefits

and

occupations

On

the

other

hand

the

naive

Bayes

classifier

distributed

classes

more

evenly

across

the

topics

This

behavior

suggests

areas

for

future

improvement

Most

importantly

we

observed

a

strong

correlation

among

the

three

most

frequent

classes

among

the

human

judges

-LRB-

for

instance

there

was

correlation

between

benefits

and

occupations

-RRB-

This

suggests

that

improving

the

clustering

to

produce

topics

that

were

more

nearly

orthogonal

might

improve

performance

Many

developers

and

maintainers

of

digital

libraries

share

the

basic

problem

pursued

here

Given

increasingly

large

complex

bodies

of

data

how

may

we

improve

access

to

collections

without

incurring

extraordinary

cost

and

while

also

keeping

systems

receptive

to

changes

in

content

over

time

Data

mining

and

machine

learning

methods

hold

a

great

deal

of

promise

with

respect

to

this

problem

Empirical

methods

of

knowledge

discovery

can

aid

in

the

organization

and

retrieval

of

information

As

we

have

argued

in

this

paper

these

methods

may

also

be

brought

to

bear

on

the

design

and

implementation

of

advanced

user

interfaces

This

study

explored

a

hybrid

technique

for

aiding

information

architects

as

they

implement

dynamic

interfaces

such

as

the

relation

browser

Our

approach

combines

unsupervised

learning

techniques

applied

to

a

focused

subset

of

the

BLS

website

The

goal

of

this

initial

stage

is

to

discover

the

most

basic

and

far-reaching

topics

in

the

collection

Based

jobs

prices

spending

costs

Human

Classifications

jobs

prices

spending

costs

Machine

Classifications

Figure

Distribution

of

Classes

Across

Documents

on

a

statistical

model

of

these

topics

the

second

phase

of

our

approach

uses

supervised

learning

-LRB-

in

particular

a

naive

Bayes

classifier

trained

on

individual

words

-RRB-

to

assign

topical

relations

to

the

remaining

documents

in

the

collection

In

the

study

reported

here

this

approach

has

demon-strated

promise

In

its

favor

our

approach

is

highly

scalable

It

also

appears

to

give

fairly

good

results

Comparing

three

modes

of

document

representation

full-text

title

only

and

keyword

we

found

accuracy

as

measured

by

collocation

of

documents

with

identical

subject

headings

While

it

is

not

surprising

that

editor-generated

keywords

should

give

strong

evidence

for

such

learning

their

superiority

over

full-text

and

titles

was

dramatic

suggesting

that

even

a

small

amount

of

metadata

can

be

very

useful

for

data

mining

However

we

also

found

evidence

that

learning

topics

from

a

subset

of

the

collection

may

lead

to

overfitted

models

After

clustering

Editor

's

Desk

documents

into

categories

we

fitted

a

naive

Bayes

classifier

to

categorize

the

remaining

documents

in

the

collection

While

we

saw

fairly

good

results

-LRB-

classification

accuracy

of

with

respect

to

a

small

sample

of

human

judges

-RRB-

this

experiment

forced

us

to

reconsider

the

quality

of

the

topics

learned

by

clustering

The

high

correlation

among

human

judgments

in

our

sample

suggests

that

the

topics

discovered

by

analysis

of

the

Editor

's

Desk

were

not

independent

While

we

do

not

desire

mutually

exclusive

categories

in

our

setting

we

do

desire

independence

among

the

topics

we

model

Overall

then

the

techniques

described

here

provide

an

encouraging

start

to

our

work

on

acquiring

subject

metadata

for

dynamic

interfaces

automatically

It

also

suggests

that

a

more

sophisticated

modeling

approach

might

yield

better

results

in

the

future

In

upcoming

work

we

will

experiment

with

streamlining

the

two-phase

technique

described

here

Instead

of

clustering

documents

to

find

topics

and

then

fitting

a

model

to

the

learned

clusters

our

goal

is

to

expand

the

unsupervised

portion

of

our

analysis

beyond

a

narrow

subset

of

the

collection

such

as

The

Editor

's

Desk

In

current

work

we

have

defined

algorithms

to

identify

documents

likely

to

help

the

topic

discovery

task

Supplied

with

a

more

comprehensive

training

set

we

hope

to

experiment

with

model-based

clustering

which

combines

the

clustering

and

classification

processes

into

a

single

modeling

procedure

Topic

discovery

and

document

classification

have

long

been

recognized

as

fundamental

problems

in

information

retrieval

and

other

forms

of

text

mining

What

is

increasingly

clear

however

as

digital

libraries

grow

in

scope

and

complexity

is

the

applicability

of

these

techniques

to

problems

at

the

front-end

of

systems

such

as

information

architecture

and

interface

design

Finally

then

in

future

work

we

will

build

on

the

user

studies

undertaken

by

Marchionini

and

Brunk

in

efforts

to

evaluate

the

utility

of

automatically

populated

dynamic

interfaces

for

the

users

of

digital

libraries

-LSB-

-RSB-

A.

Agresti

An

Introduction

to

Categorical

Data

Analysis

Wiley

New

York

-LSB-

-RSB-

C.

Ahlberg

C.

Williamson

and

B.

Shneiderman

Dynamic

queries

for

information

exploration

an

implementation

and

evaluation

In

Proceedings

of

the

SIGCHI

conference

on

Human

factors

in

computing

systems

pages

-LSB-

-RSB-

R.

Baeza-Yates

and

B.

Ribeiro-Neto

Modern

Information

Retrieval

ACM

Press

-LSB-

-RSB-

A.

Blum

and

T.

Mitchell

Combining

labeled

and

unlabeled

data

with

co-training

In

Proceedings

of

the

eleventh

annual

conference

on

Computational

learning

theory

pages

ACM

Press

-LSB-

-RSB-

H.

Chen

and

S.

Dumais

Hierarchical

classification

of

web

content

In

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

information

retrieval

pages

-LSB-

-RSB-

M.

Efron

G.

Marchionini

and

J.

Zhang

Implications

of

the

recursive

representation

problem

for

automatic

concept

identification

in

on-line

governmental

information

In

Proceedings

of

the

ASIST

Special

Interest

Group

on

Classification

Research

-LRB-

ASIST

SIG-CR

-RRB-

-LSB-

-RSB-

C.

Fraley

and

A.

E.

Raftery

How

many

clusters

which

clustering

method

answers

via

model-based

cluster

analysis

The

Computer

Journal

-LRB-

-RRB-

-LSB-

-RSB-

A.

K.

Jain

M.

N.

Murty

and

P.

J.

Flynn

Data

clustering

a

review

ACM

Computing

Surveys

-LRB-

-RRB-

September

-LSB-

-RSB-

T.

Joachims

A

probabilistic

analysis

of

the

Rocchio

algorithm

with

TFIDF

for

text

categorization

In

D.

H.

Fisher

editor

Proceedings

of

International

Conference

on

Machine

Learning

pages

Nashville

US

Morgan

Kaufmann

Publishers

San

Francisco

US

-LSB-

-RSB-

T.

Joachims

Text

categorization

with

support

vector

machines

learning

with

many

relevant

features

In

C.

N

edellec

and

C.

Rouveirol

editors

Proceedings

of

European

Conference

on

Machine

Learning

pages

Chemnitz

DE

Springer

Verlag

Heidelberg

DE

-LSB-

-RSB-

I.

T.

Jolliffe

Principal

Component

Analysis

Springer

edition

-LSB-

-RSB-

L.

Kaufman

and

P.

J.

Rosseeuw

Finding

Groups

in

Data

an

Introduction

to

Cluster

Analysis

Wiley

-LSB-

-RSB-

G.

Marchionini

and

B.

Brunk

Toward

a

general

relation

browser

a

GUI

for

information

architects

Journal

of

Digital

Information

-LRB-

-RRB-

-LSB-

-RSB-

A.

K.

McCallum

Bow

A

toolkit

for

statistical

language

modeling

text

retrieval

classification

and

clustering

-LSB-

-RSB-

T.

Mitchell

Machine

Learning

McGraw

Hill

-LSB-

-RSB-

E.

Rasmussen

Clustering

algorithms

In

W.

B.

Frakes

and

R.

Baeza-Yates

editors

Information

Retrieval

Data

Structures

and

Algorithms

pages

Prentice

Hall

-LSB-

-RSB-

R.

Tibshirani

G.

Walther

and

T.

Hastie

Estimating

the

number

of

clusters

in

a

dataset

via

the

gap

statistic

-LSB-

-RSB-

V.

N.

Vapnik

The

Nature

of

Statistical

Learning

Theory

Springer

B I O B I O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

bls
clustering
document representation
topic discovery
interface design
machine learning
information architecture
document classification
digital libraries