Automatic

Extraction

of

Titles

from

General

Documents

using

Machine

Learning

In

this

paper

we

propose

a

machine

learning

approach

to

title

extraction

from

general

documents

By

general

documents

we

mean

documents

that

can

belong

to

any

one

of

a

number

of

specific

genres

including

presentations

book

chapters

technical

papers

brochures

reports

and

letters

Previously

methods

have

been

proposed

mainly

for

title

extraction

from

research

papers

It

has

not

been

clear

whether

it

could

be

possible

to

conduct

automatic

title

extraction

from

general

documents

As

a

case

study

we

consider

extraction

from

Office

including

Word

and

PowerPoint

In

our

approach

we

annotate

titles

in

sample

documents

-LRB-

for

Word

and

PowerPoint

respectively

-RRB-

and

take

them

as

training

data

train

machine

learning

models

and

perform

title

extraction

using

the

trained

models

Our

method

is

unique

in

that

we

mainly

utilize

formatting

information

such

as

font

size

as

features

in

the

models

It

turns

out

that

the

use

of

formatting

information

can

lead

to

quite

accurate

extraction

from

general

documents

Precision

and

recall

for

title

extraction

from

Word

is

and

respectively

and

precision

and

recall

for

title

extraction

from

PowerPoint

is

and

respectively

in

an

experiment

on

intranet

data

Other

important

new

findings

in

this

work

include

that

we

can

train

models

in

one

domain

and

apply

them

to

another

domain

and

more

surprisingly

we

can

even

train

models

in

one

language

and

apply

them

to

another

language

Moreover

we

can

significantly

improve

search

ranking

results

in

document

retrieval

by

using

the

extracted

titles

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval

Search

Process

H.

-LSB-

Information

Systems

Applications

-RSB-

Office

Automation

Word

processing

D.

-LSB-

Software

Engineering

-RSB-

Metrics

complexity

measures

performance

measures

Algorithms

Experimentation

Performance

Metadata

of

documents

is

useful

for

many

kinds

of

document

processing

such

as

search

browsing

and

filtering

Ideally

metadata

is

defined

by

the

authors

of

documents

and

is

then

used

by

various

systems

However

people

seldom

define

document

metadata

by

themselves

even

when

they

have

convenient

metadata

definition

tools

-LSB-

-RSB-

Thus

how

to

automatically

extract

metadata

from

the

bodies

of

documents

turns

out

to

be

an

important

research

issue

Methods

for

performing

the

task

have

been

proposed

However

the

focus

was

mainly

on

extraction

from

research

papers

For

instance

Han

et

al.

-LSB-

-RSB-

proposed

a

machine

learning

based

method

to

conduct

extraction

from

research

papers

They

formalized

the

problem

as

that

of

classification

and

employed

Support

Vector

Machines

as

the

classifier

They

mainly

used

linguistic

features

in

the

model

In

this

paper

we

consider

metadata

extraction

from

general

documents

By

general

documents

we

mean

documents

that

may

belong

to

any

one

of

a

number

of

specific

genres

General

documents

are

more

widely

available

in

digital

libraries

intranets

and

the

internet

and

thus

investigation

on

extraction

from

them

is

The

work

was

conducted

when

the

first

author

was

visiting

Microsoft

Research

Asia

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

JCDL

June

Denver

Colorado

USA

Copyright

ACM

sorely

needed

Research

papers

usually

have

well-formed

styles

and

noticeable

characteristics

In

contrast

the

styles

of

general

documents

can

vary

greatly

It

has

not

been

clarified

whether

a

machine

learning

based

approach

can

work

well

for

this

task

There

are

many

types

of

metadata

title

author

date

of

creation

etc.

As

a

case

study

we

consider

title

extraction

in

this

paper

General

documents

can

be

in

many

different

file

formats

Microsoft

Office

PDF

-LRB-

PS

-RRB-

etc.

As

a

case

study

we

consider

extraction

from

Office

including

Word

and

PowerPoint

We

take

a

machine

learning

approach

We

annotate

titles

in

sample

documents

-LRB-

for

Word

and

PowerPoint

respectively

-RRB-

and

take

them

as

training

data

to

train

several

types

of

models

and

perform

title

extraction

using

any

one

type

of

the

trained

models

In

the

models

we

mainly

utilize

formatting

information

such

as

font

size

as

features

We

employ

the

following

models

Maximum

Entropy

Model

Perceptron

with

Uneven

Margins

Maximum

Entropy

Markov

Model

and

Voted

Perceptron

In

this

paper

we

also

investigate

the

following

three

problems

which

did

not

seem

to

have

been

examined

previously

-LRB-

-RRB-

Comparison

between

models

among

the

models

above

which

model

performs

best

for

title

extraction

-LRB-

-RRB-

Generality

of

model

whether

it

is

possible

to

train

a

model

on

one

domain

and

apply

it

to

another

domain

and

whether

it

is

possible

to

train

a

model

in

one

language

and

apply

it

to

another

language

-LRB-

-RRB-

Usefulness

of

extracted

titles

whether

extracted

titles

can

improve

document

processing

such

as

search

Experimental

results

indicate

that

our

approach

works

well

for

title

extraction

from

general

documents

Our

method

can

significantly

outperform

the

baselines

one

that

always

uses

the

first

lines

as

titles

and

the

other

that

always

uses

the

lines

in

the

largest

font

sizes

as

titles

Precision

and

recall

for

title

extraction

from

Word

are

and

respectively

and

precision

and

recall

for

title

extraction

from

PowerPoint

are

and

respectively

It

turns

out

that

the

use

of

format

features

is

the

key

to

successful

title

extraction

-LRB-

-RRB-

We

have

observed

that

Perceptron

based

models

perform

better

in

terms

of

extraction

accuracies

-LRB-

-RRB-

We

have

empirically

verified

that

the

models

trained

with

our

approach

are

generic

in

the

sense

that

they

can

be

trained

on

one

domain

and

applied

to

another

and

they

can

be

trained

in

one

language

and

applied

to

another

-LRB-

-RRB-

We

have

found

that

using

the

extracted

titles

we

can

significantly

improve

precision

of

document

retrieval

-LRB-

by

-RRB-

We

conclude

that

we

can

indeed

conduct

reliable

title

extraction

from

general

documents

and

use

the

extracted

results

to

improve

real

applications

The

rest

of

the

paper

is

organized

as

follows

In

section

we

introduce

related

work

and

in

section

we

explain

the

motivation

and

problem

setting

of

our

work

In

section

we

describe

our

method

of

title

extraction

and

in

section

we

describe

our

method

of

document

retrieval

using

extracted

titles

Section

gives

our

experimental

results

We

make

concluding

remarks

in

section

Methods

have

been

proposed

for

performing

automatic

metadata

extraction

from

documents

however

the

main

focus

was

on

extraction

from

research

papers

The

proposed

methods

fall

into

two

categories

the

rule

based

approach

and

the

machine

learning

based

approach

Giuffrida

et

al.

-LSB-

-RSB-

for

instance

developed

a

rule-based

system

for

automatically

extracting

metadata

from

research

papers

in

Postscript

They

used

rules

like

titles

are

usually

located

on

the

upper

portions

of

the

first

pages

and

they

are

usually

in

the

largest

font

sizes

Liddy

et

al.

-LSB-

-RSB-

and

Yilmazel

el

al.

-LSB-

-RSB-

performed

metadata

extraction

from

educational

materials

using

rule-based

natural

language

processing

technologies

Mao

et

al.

-LSB-

-RSB-

also

conducted

automatic

metadata

extraction

from

research

papers

using

rules

on

formatting

information

The

rule-based

approach

can

achieve

high

performance

However

it

also

has

disadvantages

It

is

less

adaptive

and

robust

when

compared

with

the

machine

learning

approach

Han

et

al.

-LSB-

-RSB-

for

instance

conducted

metadata

extraction

with

the

machine

learning

approach

They

viewed

the

problem

as

that

of

classifying

the

lines

in

a

document

into

the

categories

of

metadata

and

proposed

using

Support

Vector

Machines

as

the

classifier

They

mainly

used

linguistic

information

as

features

They

reported

high

extraction

accuracy

from

research

papers

in

terms

of

precision

and

recall

Information

Extraction

Metadata

extraction

can

be

viewed

as

an

application

of

information

extraction

in

which

given

a

sequence

of

instances

we

identify

a

subsequence

that

represents

information

in

which

we

are

interested

Hidden

Markov

Model

-LSB-

-RSB-

Maximum

Entropy

Model

-LSB-

-RSB-

Maximum

Entropy

Markov

Model

-LSB-

-RSB-

Support

Vector

Machines

-LSB-

-RSB-

Conditional

Random

Field

-LSB-

-RSB-

and

Voted

Perceptron

-LSB-

-RSB-

are

widely

used

information

extraction

models

Information

extraction

has

been

applied

for

instance

to

part-of-speech

tagging

-LSB-

-RSB-

named

entity

recognition

-LSB-

-RSB-

and

table

extraction

-LSB-

-RSB-

Search

Using

Title

Information

Title

information

is

useful

for

document

retrieval

In

the

system

Citeseer

for

instance

Giles

et

al.

managed

to

extract

titles

from

research

papers

and

make

use

of

the

extracted

titles

in

metadata

search

of

papers

-LSB-

-RSB-

In

web

search

the

title

fields

-LRB-

i.e.

file

properties

-RRB-

and

anchor

texts

of

web

pages

-LRB-

HTML

documents

-RRB-

can

be

viewed

as

titles

of

the

pages

-LSB-

-RSB-

Many

search

engines

seem

to

utilize

them

for

web

page

retrieval

-LSB-

-RSB-

Zhang

et

al.

found

that

web

pages

with

well-defined

metadata

are

more

easily

retrieved

than

those

without

well-defined

metadata

-LSB-

-RSB-

To

the

best

of

our

knowledge

no

research

has

been

conducted

on

using

extracted

titles

from

general

documents

-LRB-

e.g.

Office

documents

-RRB-

for

search

of

the

documents

We

consider

the

issue

of

automatically

extracting

titles

from

general

documents

By

general

documents

we

mean

documents

that

belong

to

one

of

any

number

of

specific

genres

The

documents

can

be

presentations

books

book

chapters

technical

papers

brochures

reports

memos

specifications

letters

announcements

or

resumes

General

documents

are

more

widely

available

in

digital

libraries

intranets

and

internet

and

thus

investigation

on

title

extraction

from

them

is

sorely

needed

Figure

shows

an

estimate

on

distributions

of

file

formats

on

intranet

and

internet

-LSB-

-RSB-

Office

and

PDF

are

the

main

file

formats

on

the

intranet

Even

on

the

internet

the

documents

in

the

formats

are

still

not

negligible

given

its

extremely

large

size

In

this

paper

without

loss

of

generality

we

take

Office

documents

as

an

example

Figure

Distributions

of

file

formats

in

internet

and

intranet

For

Office

documents

users

can

define

titles

as

file

properties

using

a

feature

provided

by

Office

We

found

in

an

experiment

however

that

users

seldom

use

the

feature

and

thus

titles

in

file

properties

are

usually

very

inaccurate

That

is

to

say

titles

in

file

properties

are

usually

inconsistent

with

the

true

titles

in

the

file

bodies

that

are

created

by

the

authors

and

are

visible

to

readers

We

collected

Word

and

PowerPoint

documents

from

an

intranet

and

the

internet

and

examined

how

many

titles

in

the

file

properties

are

correct

We

found

that

surprisingly

the

accuracy

was

only

-LRB-

cf.

Section

for

details

-RRB-

A

number

of

reasons

can

be

considered

For

example

if

one

creates

a

new

file

by

copying

an

old

file

then

the

file

property

of

the

new

file

will

also

be

copied

from

the

old

file

In

another

experiment

we

found

that

Google

uses

the

titles

in

file

properties

of

Office

documents

in

search

and

browsing

but

the

titles

are

not

very

accurate

We

created

queries

to

search

Word

and

PowerPoint

documents

and

examined

the

top

results

of

each

query

returned

by

Google

We

found

that

nearly

all

the

titles

presented

in

the

search

results

were

from

the

file

properties

of

the

documents

However

only

of

them

were

correct

Actually

true

titles

usually

exist

at

the

beginnings

of

the

bodies

of

documents

If

we

can

accurately

extract

the

titles

from

the

bodies

of

documents

then

we

can

exploit

reliable

title

information

in

document

processing

This

is

exactly

the

problem

we

address

in

this

paper

More

specifically

given

a

Word

document

we

are

to

extract

the

title

from

the

top

region

of

the

first

page

Given

a

PowerPoint

document

we

are

to

extract

the

title

from

the

first

slide

A

title

sometimes

consists

of

a

main

title

and

one

or

two

subtitles

We

only

consider

extraction

of

the

main

title

As

baselines

for

title

extraction

we

use

that

of

always

using

the

first

lines

as

titles

and

that

of

always

using

the

lines

with

largest

font

sizes

as

titles

Figure

Title

extraction

from

Word

document

Figure

Title

extraction

from

PowerPoint

document

Next

we

define

a

specification

for

human

judgments

in

title

data

annotation

The

annotated

data

will

be

used

in

training

and

testing

of

the

title

extraction

methods

Summary

of

the

specification

The

title

of

a

document

should

be

identified

on

the

basis

of

common

sense

if

there

is

no

difficulty

in

the

identification

However

there

are

many

cases

in

which

the

identification

is

not

easy

There

are

some

rules

defined

in

the

specification

that

guide

identification

for

such

cases

The

rules

include

a

title

is

usually

in

consecutive

lines

in

the

same

format

a

document

can

have

no

title

titles

in

images

are

not

considered

a

title

should

not

contain

words

like

draft

whitepaper

etc

if

it

is

difficult

to

determine

which

is

the

title

select

the

one

in

the

largest

font

size

and

if

it

is

still

difficult

to

determine

which

is

the

title

select

the

first

candidate

-LRB-

The

specification

covers

all

the

cases

we

have

encountered

in

data

annotation

-RRB-

Figures

and

show

examples

of

Office

documents

from

which

we

conduct

title

extraction

In

Figure

Differences

in

API

Implementations

among

Windows

Operating

Systems

is

the

title

of

the

Word

document

Microsoft

Windows

on

the

top

of

this

page

is

a

picture

and

thus

is

ignored

In

Figure

Building

Competitive

Advantages

through

an

Agile

Infrastructure

is

the

title

of

the

PowerPoint

document

We

have

developed

a

tool

for

annotation

of

titles

by

human

annotators

Figure

shows

a

snapshot

of

the

tool

Figure

Title

annotation

tool

Title

extraction

based

on

machine

learning

consists

of

training

and

extraction

The

same

pre-processing

step

occurs

before

training

and

extraction

During

pre-processing

from

the

top

region

of

the

first

page

of

a

Word

document

or

the

first

slide

of

a

PowerPoint

document

a

number

of

units

for

processing

are

extracted

If

a

line

-LRB-

lines

are

separated

by

return

symbols

-RRB-

only

has

a

single

format

then

the

line

will

become

a

unit

If

a

line

has

several

parts

and

each

of

them

has

its

own

format

then

each

part

will

become

a

unit

Each

unit

will

be

treated

as

an

instance

in

learning

A

unit

contains

not

only

content

information

-LRB-

linguistic

information

-RRB-

but

also

formatting

information

The

input

to

pre-processing

is

a

document

and

the

output

of

pre-processing

is

a

sequence

of

units

-LRB-

instances

-RRB-

Figure

shows

the

units

obtained

from

the

document

in

Figure

Figure

Example

of

units

In

learning

the

input

is

sequences

of

units

where

each

sequence

corresponds

to

a

document

We

take

labeled

units

-LRB-

labeled

as

or

other

-RRB-

in

the

sequences

as

training

data

and

construct

models

for

identifying

whether

a

unit

is

or

other

We

employ

four

types

of

models

Perceptron

Maximum

Entropy

-LRB-

ME

-RRB-

Perceptron

Markov

Model

-LRB-

PMM

-RRB-

and

Maximum

Entropy

Markov

Model

-LRB-

MEMM

-RRB-

In

extraction

the

input

is

a

sequence

of

units

from

one

document

We

employ

one

type

of

model

to

identify

whether

a

unit

is

or

other

We

then

extract

units

from

the

unit

labeled

with

to

the

unit

labeled

with

The

result

is

the

extracted

title

of

the

document

The

unique

characteristic

of

our

approach

is

that

we

mainly

utilize

formatting

information

for

title

extraction

Our

assumption

is

that

although

general

documents

vary

in

styles

their

formats

have

certain

patterns

and

we

can

learn

and

utilize

the

patterns

for

title

extraction

This

is

in

contrast

to

the

work

by

Han

et

al.

in

which

only

linguistic

features

are

used

for

extraction

from

research

papers

Models

The

four

models

actually

can

be

considered

in

the

same

metadata

extraction

framework

That

is

why

we

apply

them

together

to

our

current

problem

Each

input

is

a

sequence

of

instances

k

x

x

x

L

together

with

a

sequence

of

labels

k

y

y

y

L

i

x

and

i

y

represents

an

instance

and

its

label

respectively

-LRB-

k

i

L

-RRB-

Recall

that

an

instance

here

represents

a

unit

A

label

represents

or

other

Here

k

is

the

number

of

units

in

a

document

In

learning

we

train

a

model

which

can

be

generally

denoted

as

a

conditional

probability

distribution

-RRB-

-LRB-

k

k

X

X

Y

Y

P

L

L

where

i

X

and

i

Y

denote

random

variables

taking

instance

i

x

and

label

i

y

as

values

respectively

-LRB-

k

i

L

-RRB-

Learning

Tool

Extraction

Tool

nk

n

n

k

n

n

k

k

k

k

y

y

y

x

x

x

y

y

y

x

x

x

y

y

y

x

x

x

L

L

L

L

L

L

L

L

-RRB-

-LRB-

max

arg

mk

m

mk

m

x

x

y

y

P

L

L

-RRB-

-LRB-

k

k

X

X

Y

Y

P

L

L

Conditional

Distribution

mk

m

m

x

x

x

L

Figure

Metadata

extraction

model

We

can

make

assumptions

about

the

general

model

in

order

to

make

it

simple

enough

for

training

For

example

we

can

assume

that

k

Y

Y

L

are

independent

of

each

other

given

k

X

X

L

Thus

we

have

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

k

k

k

k

X

Y

P

X

Y

P

X

X

Y

Y

P

L

L

L

In

this

way

we

decompose

the

model

into

a

number

of

classifiers

We

train

the

classifiers

locally

using

the

labeled

data

As

the

classifier

we

employ

the

Perceptron

or

Maximum

Entropy

model

We

can

also

assume

that

the

first

order

Markov

property

holds

for

k

Y

Y

L

given

k

X

X

L

Thus

we

have

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

k

k

k

k

k

X

Y

Y

P

X

Y

P

X

X

Y

Y

P

L

L

L

Again

we

obtain

a

number

of

classifiers

However

the

classifiers

are

conditioned

on

the

previous

label

When

we

employ

the

Percepton

or

Maximum

Entropy

model

as

a

classifier

the

models

become

a

Percepton

Markov

Model

or

Maximum

Entropy

Markov

Model

respectively

That

is

to

say

the

two

models

are

more

precise

In

extraction

given

a

new

sequence

of

instances

we

resort

to

one

of

the

constructed

models

to

assign

a

sequence

of

labels

to

the

sequence

of

instances

i.e.

perform

extraction

For

Perceptron

and

ME

we

assign

labels

locally

and

combine

the

results

globally

later

using

heuristics

Specifically

we

first

identify

the

most

likely

Then

we

find

the

most

likely

within

three

units

after

the

Finally

we

extract

as

a

title

the

units

between

the

and

the

For

PMM

and

MEMM

we

employ

the

Viterbi

algorithm

to

find

the

globally

optimal

label

sequence

In

this

paper

for

Perceptron

we

actually

employ

an

improved

variant

of

it

called

Perceptron

with

Uneven

Margin

-LSB-

-RSB-

This

version

of

Perceptron

can

work

well

especially

when

the

number

of

positive

instances

and

the

number

of

negative

instances

differ

greatly

which

is

exactly

the

case

in

our

problem

We

also

employ

an

improved

version

of

Perceptron

Markov

Model

in

which

the

Perceptron

model

is

the

so-called

Voted

Perceptron

-LSB-

-RSB-

In

addition

in

training

the

parameters

of

the

model

are

updated

globally

rather

than

locally

Features

There

are

two

types

of

features

format

features

and

linguistic

features

We

mainly

use

the

former

The

features

are

used

for

both

the

title-begin

and

the

title-end

classifiers

Format

Features

Font

Size

There

are

four

binary

features

that

represent

the

normalized

font

size

of

the

unit

-LRB-

recall

that

a

unit

has

only

one

type

of

font

-RRB-

If

the

font

size

of

the

unit

is

the

largest

in

the

document

then

the

first

feature

will

be

otherwise

If

the

font

size

is

the

smallest

in

the

document

then

the

fourth

feature

will

be

otherwise

If

the

font

size

is

above

the

average

font

size

and

not

the

largest

in

the

document

then

the

second

feature

will

be

otherwise

If

the

font

size

is

below

the

average

font

size

and

not

the

smallest

the

third

feature

will

be

otherwise

It

is

necessary

to

conduct

normalization

on

font

sizes

For

example

in

one

document

the

largest

font

size

might

be

pt

while

in

another

the

smallest

one

might

be

pt

Boldface

This

binary

feature

represents

whether

or

not

the

current

unit

is

in

boldface

Alignment

There

are

four

binary

features

that

respectively

represent

the

location

of

the

current

unit

left

center

right

and

unknown

alignment

The

following

format

features

with

respect

to

context

play

an

important

role

in

title

extraction

Empty

Neighboring

Unit

There

are

two

binary

features

that

represent

respectively

whether

or

not

the

previous

unit

and

the

current

unit

are

blank

lines

Font

Size

Change

There

are

two

binary

features

that

represent

respectively

whether

or

not

the

font

size

of

the

previous

unit

and

the

font

size

of

the

next

unit

differ

from

that

of

the

current

unit

Alignment

Change

There

are

two

binary

features

that

represent

respectively

whether

or

not

the

alignment

of

the

previous

unit

and

the

alignment

of

the

next

unit

differ

from

that

of

the

current

one

Same

Paragraph

There

are

two

binary

features

that

represent

respectively

whether

or

not

the

previous

unit

and

the

next

unit

are

in

the

same

paragraph

as

the

current

unit

Linguistic

Features

The

linguistic

features

are

based

on

key

words

Positive

Word

This

binary

feature

represents

whether

or

not

the

current

unit

begins

with

one

of

the

positive

words

The

positive

words

include

title

subject

subject

line

For

example

in

some

documents

the

lines

of

titles

and

authors

have

the

same

formats

However

if

lines

begin

with

one

of

the

positive

words

then

it

is

likely

that

they

are

title

lines

Negative

Word

This

binary

feature

represents

whether

or

not

the

current

unit

begins

with

one

of

the

negative

words

The

negative

words

include

To

By

created

by

updated

by

etc.

There

are

more

negative

words

than

positive

words

The

above

linguistic

features

are

language

dependent

Word

Count

A

title

should

not

be

too

long

We

heuristically

create

four

intervals

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

and

-LSB-

-RRB-

and

define

one

feature

for

each

interval

If

the

number

of

words

in

a

title

falls

into

an

interval

then

the

corresponding

feature

will

be

otherwise

Ending

Character

This

feature

represents

whether

the

unit

ends

with

or

other

special

characters

A

title

usually

does

not

end

with

such

a

character

We

describe

our

method

of

document

retrieval

using

extracted

titles

Typically

in

information

retrieval

a

document

is

split

into

a

number

of

fields

including

body

title

and

anchor

text

A

ranking

function

in

search

can

use

different

weights

for

different

fields

of

the

document

Also

titles

are

typically

assigned

high

weights

indicating

that

they

are

important

for

document

retrieval

As

explained

previously

our

experiment

has

shown

that

a

significant

number

of

documents

actually

have

incorrect

titles

in

the

file

properties

and

thus

in

addition

of

using

them

we

use

the

extracted

titles

as

one

more

field

of

the

document

By

doing

this

we

attempt

to

improve

the

overall

precision

In

this

paper

we

employ

a

modification

of

that

allows

field

weighting

-LSB-

-RSB-

As

fields

we

make

use

of

body

title

extracted

title

and

anchor

First

for

each

term

in

the

query

we

count

the

term

frequency

in

each

field

of

the

document

each

field

frequency

is

then

weighted

according

to

the

corresponding

weight

parameter

f

tf

f

t

tf

w

wtf

Similarly

we

compute

the

document

length

as

a

weighted

sum

of

lengths

of

each

field

Average

document

length

in

the

corpus

becomes

the

average

of

all

weighted

document

lengths

f

f

f

dl

w

wdl

In

our

experiments

we

used

b

k

Weight

for

content

was

title

was

anchor

was

and

extracted

title

was

We

used

two

data

sets

in

our

experiments

First

we

downloaded

and

randomly

selected

Word

documents

and

PowerPoint

documents

from

an

intranet

of

Microsoft

We

call

it

MS

hereafter

Second

we

downloaded

and

randomly

selected

Word

and

PowerPoint

documents

from

the

DotGov

and

DotCom

domains

on

the

internet

respectively

Figure

shows

the

distributions

of

the

genres

of

the

documents

We

see

that

the

documents

are

indeed

general

documents

as

we

define

them

Figure

Distributions

of

document

genres

Third

a

data

set

in

Chinese

was

also

downloaded

from

the

internet

It

includes

Word

documents

and

PowerPoint

documents

in

Chinese

We

manually

labeled

the

titles

of

all

the

documents

on

the

basis

of

our

specification

Not

all

the

documents

in

the

two

data

sets

have

titles

Table

shows

the

percentages

of

the

documents

having

titles

We

see

that

DotCom

and

DotGov

have

more

PowerPoint

documents

with

titles

than

MS.

This

might

be

because

PowerPoint

documents

published

on

the

internet

are

more

formal

than

those

on

the

intranet

Table

The

portion

of

documents

with

titles

Domain

Type

MS

DotCom

DotGov

Word

PowerPoint

In

our

experiments

we

conducted

evaluations

on

title

extraction

in

terms

of

precision

recall

and

F-measure

The

evaluation

measures

are

defined

as

follows

Precision

P

A

-LRB-

A

B

-RRB-

Recall

R

A

-LRB-

A

C

-RRB-

F-measure

-LRB-

P

R

-RRB-

Here

A

B

C

and

D

are

numbers

of

documents

as

those

defined

in

Table

Table

Contingence

table

with

regard

to

title

extraction

Is

title

Is

not

title

Extracted

A

B

Not

extracted

C

D

Baselines

We

test

the

accuracies

of

the

two

baselines

described

in

section

They

are

denoted

as

largest

font

size

and

first

line

respectively

Accuracy

of

Titles

in

File

Properties

We

investigate

how

many

titles

in

the

file

properties

of

the

documents

are

reliable

We

view

the

titles

annotated

by

humans

as

true

titles

and

test

how

many

titles

in

the

file

properties

can

approximately

match

with

the

true

titles

We

use

Edit

Distance

to

conduct

the

approximate

match

-LRB-

Approximate

match

is

only

used

in

this

evaluation

-RRB-

This

is

because

sometimes

human

annotated

titles

can

be

slightly

different

from

the

titles

in

file

properties

on

the

surface

e.g.

contain

extra

spaces

-RRB-

Given

string

A

and

string

B

if

-LRB-

-LRB-

D

-RRB-

or

-LRB-

D

-LRB-

La

Lb

-RRB-

-RRB-

-RRB-

then

string

A

string

B

D

Edit

Distance

between

string

A

and

string

B

La

length

of

string

A

Lb

length

of

string

B

t

t

n

N

wtf

avwdl

wdl

b

b

k

k

wtf

F

BM

-RRB-

log

-LRB-

-RRB-

-RRB-

-LRB-

-LRB-

-RRB-

-LRB-

Table

Accuracies

of

titles

in

file

properties

File

Type

Domain

Precision

Recall

MS

DotCom

Word

DotGov

MS

DotCom

PowerPoint

DotGov

Comparison

with

Baselines

We

conducted

title

extraction

from

the

first

data

set

-LRB-

Word

and

PowerPoint

in

MS

-RRB-

As

the

model

we

used

Perceptron

We

conduct

cross

validation

Thus

all

the

results

reported

here

are

those

averaged

over

trials

Tables

and

show

the

results

We

see

that

Perceptron

significantly

outperforms

the

baselines

In

the

evaluation

we

use

exact

matching

between

the

true

titles

annotated

by

humans

and

the

extracted

titles

Table

Accuracies

of

title

extraction

with

Word

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

We

see

that

the

machine

learning

approach

can

achieve

good

performance

in

title

extraction

For

Word

documents

both

precision

and

recall

of

the

approach

are

percent

higher

than

those

of

the

baselines

For

PowerPoint

both

precision

and

recall

of

the

approach

are

percent

higher

than

those

of

the

baselines

We

conduct

significance

tests

The

results

are

shown

in

Table

Here

Largest

denotes

the

baseline

of

using

the

largest

font

size

First

denotes

the

baseline

of

using

the

first

line

The

results

indicate

that

the

improvements

of

machine

learning

over

baselines

are

statistically

significant

-LRB-

in

the

sense

p-value

-RRB-

Table

Sign

test

results

Documents

Type

Sign

test

between

p-value

Perceptron

vs.

Largest

Word

Perceptron

vs.

First

Perceptron

vs.

Largest

PowerPoint

Perceptron

vs.

First

We

see

from

the

results

that

the

two

baselines

can

work

well

for

title

extraction

suggesting

that

font

size

and

position

information

are

most

useful

features

for

title

extraction

However

it

is

also

obvious

that

using

only

these

two

features

is

not

enough

There

are

cases

in

which

all

the

lines

have

the

same

font

size

-LRB-

i.e.

the

largest

font

size

-RRB-

or

cases

in

which

the

lines

with

the

largest

font

size

only

contain

general

descriptions

like

Confidential

White

paper

etc.

For

those

cases

the

largest

font

size

method

can

not

work

well

For

similar

reasons

the

first

line

method

alone

can

not

work

well

either

With

the

combination

of

different

features

-LRB-

evidence

in

title

judgment

-RRB-

Perceptron

can

outperform

Largest

and

First

We

investigate

the

performance

of

solely

using

linguistic

features

We

found

that

it

does

not

work

well

It

seems

that

the

format

features

play

important

roles

and

the

linguistic

features

are

supplements

Figure

An

example

Word

document

Figure

An

example

PowerPoint

document

We

conducted

an

error

analysis

on

the

results

of

Perceptron

We

found

that

the

errors

fell

into

three

categories

-LRB-

-RRB-

About

one

third

of

the

errors

were

related

to

hard

cases

In

these

documents

the

layouts

of

the

first

pages

were

difficult

to

understand

even

for

humans

Figure

and

shows

examples

-LRB-

-RRB-

Nearly

one

fourth

of

the

errors

were

from

the

documents

which

do

not

have

true

titles

but

only

contain

bullets

Since

we

conduct

extraction

from

the

top

regions

it

is

difficult

to

get

rid

of

these

errors

with

the

current

approach

-LRB-

-RRB-

Confusions

between

main

titles

and

subtitles

were

another

type

of

error

Since

we

only

labeled

the

main

titles

as

titles

the

extractions

of

both

titles

were

considered

incorrect

This

type

of

error

does

little

harm

to

document

processing

like

search

however

Comparison

between

Models

To

compare

the

performance

of

different

machine

learning

models

we

conducted

another

experiment

Again

we

perform

cross

validation

on

the

first

data

set

-LRB-

MS

-RRB-

Table

shows

the

results

of

all

the

four

models

It

turns

out

that

Perceptron

and

PMM

perform

the

best

followed

by

MEMM

and

ME

performs

the

worst

In

general

the

Markovian

models

perform

better

than

or

as

well

as

their

classifier

counterparts

This

seems

to

be

because

the

Markovian

models

are

trained

globally

while

the

classifiers

are

trained

locally

The

Perceptron

based

models

perform

better

than

the

ME

based

counterparts

This

seems

to

be

because

the

Perceptron

based

models

are

created

to

make

better

classifications

while

ME

models

are

constructed

for

better

prediction

Table

Comparison

between

different

learning

models

for

title

extraction

with

Word

Model

Precision

Recall

Perceptron

MEMM

PMM

ME

Table

Comparison

between

different

learning

models

for

title

extraction

with

PowerPoint

Model

Precision

Recall

Perceptron

MEMM

PMM

ME

Domain

Adaptation

We

apply

the

model

trained

with

the

first

data

set

-LRB-

MS

-RRB-

to

the

second

data

set

-LRB-

DotCom

and

DotGov

-RRB-

Tables

show

the

results

Table

Accuracies

of

title

extraction

with

Word

in

DotGov

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

in

DotGov

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Accuracies

of

title

extraction

with

Word

in

DotCom

Precisio

n

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Performance

of

PowerPoint

document

title

extraction

in

DotCom

Precisio

n

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

From

the

results

we

see

that

the

models

can

be

adapted

to

different

domains

well

There

is

almost

no

drop

in

accuracy

The

results

indicate

that

the

patterns

of

title

formats

exist

across

different

domains

and

it

is

possible

to

construct

a

domain

independent

model

by

mainly

using

formatting

information

Language

Adaptation

We

apply

the

model

trained

with

the

data

in

English

-LRB-

MS

-RRB-

to

the

data

set

in

Chinese

Tables

show

the

results

Table

Accuracies

of

title

extraction

with

Word

in

Chinese

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

in

Chinese

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

We

see

that

the

models

can

be

adapted

to

a

different

language

There

are

only

small

drops

in

accuracy

Obviously

the

linguistic

features

do

not

work

for

Chinese

but

the

effect

of

not

using

them

is

negligible

The

results

indicate

that

the

patterns

of

title

formats

exist

across

different

languages

From

the

domain

adaptation

and

language

adaptation

results

we

conclude

that

the

use

of

formatting

information

is

the

key

to

a

successful

extraction

from

general

documents

Search

with

Extracted

Titles

We

performed

experiments

on

using

title

extraction

for

document

retrieval

As

a

baseline

we

employed

without

using

extracted

titles

The

ranking

mechanism

was

as

described

in

Section

The

weights

were

heuristically

set

We

did

not

conduct

optimization

on

the

weights

The

evaluation

was

conducted

on

a

corpus

of

M

documents

crawled

from

the

intranet

of

Microsoft

using

evaluation

queries

obtained

from

this

intranet

's

search

engine

query

logs

queries

were

from

the

most

popular

set

while

queries

other

were

chosen

randomly

Users

were

asked

to

provide

judgments

of

the

degree

of

document

relevance

from

a

scale

of

-LRB-

meaning

detrimental

bad

fair

good

and

excellent

-RRB-

Figure

shows

the

results

In

the

chart

two

sets

of

precision

results

were

obtained

by

either

considering

good

or

excellent

documents

as

relevant

-LRB-

left

bars

with

relevance

threshold

-RRB-

or

by

considering

only

excellent

documents

as

relevant

-LRB-

right

bars

with

relevance

threshold

-RRB-

P

Reciprocal

P

Reciprocal

Anchor

Title

Body

Anchor

Title

Body

ExtractedTitle

Name

All

RelevanceThreshold

Data

Description

Figure

Search

ranking

results

Figure

shows

different

document

retrieval

results

with

different

ranking

functions

in

terms

of

precision

precision

and

reciprocal

rank

Blue

bar

including

the

fields

body

title

-LRB-

file

property

-RRB-

and

anchor

text

Purple

bar

including

the

fields

body

title

-LRB-

file

property

-RRB-

anchor

text

and

extracted

title

With

the

additional

field

of

extracted

title

included

in

the

precision

increased

from

to

or

by

Thus

it

is

safe

to

say

that

the

use

of

extracted

title

can

indeed

improve

the

precision

of

document

retrieval

In

this

paper

we

have

investigated

the

problem

of

automatically

extracting

titles

from

general

documents

We

have

tried

using

a

machine

learning

approach

to

address

the

problem

Previous

work

showed

that

the

machine

learning

approach

can

work

well

for

metadata

extraction

from

research

papers

In

this

paper

we

showed

that

the

approach

can

work

for

extraction

from

general

documents

as

well

Our

experimental

results

indicated

that

the

machine

learning

approach

can

work

significantly

better

than

the

baselines

in

title

extraction

from

Office

documents

Previous

work

on

metadata

extraction

mainly

used

linguistic

features

in

documents

while

we

mainly

used

formatting

information

It

appeared

that

using

formatting

information

is

a

key

for

successfully

conducting

title

extraction

from

general

documents

We

tried

different

machine

learning

models

including

Perceptron

Maximum

Entropy

Maximum

Entropy

Markov

Model

and

Voted

Perceptron

We

found

that

the

performance

of

the

Perceptorn

models

was

the

best

We

applied

models

constructed

in

one

domain

to

another

domain

and

applied

models

trained

in

one

language

to

another

language

We

found

that

the

accuracies

did

not

drop

substantially

across

different

domains

and

across

different

languages

indicating

that

the

models

were

generic

We

also

attempted

to

use

the

extracted

titles

in

document

retrieval

We

observed

a

significant

improvement

in

document

ranking

performance

for

search

when

using

extracted

title

information

All

the

above

investigations

were

not

conducted

in

previous

work

and

through

our

investigations

we

verified

the

generality

and

the

significance

of

the

title

extraction

approach

We

thank

Chunyu

Wei

and

Bojuan

Zhao

for

their

work

on

data

annotation

We

acknowledge

Jinzhu

Li

for

his

assistance

in

conducting

the

experiments

We

thank

Ming

Zhou

John

Chen

Jun

Xu

and

the

anonymous

reviewers

of

JCDL

for

their

valuable

comments

on

this

paper

-LSB-

-RSB-

Berger

A.

L.

Della

Pietra

S.

A.

and

Della

Pietra

V.

J

A

maximum

entropy

approach

to

natural

language

processing

Computational

Linguistics

-LSB-

-RSB-

Collins

M.

Discriminative

training

methods

for

hidden

markov

models

theory

and

experiments

with

perceptron

algorithms

In

Proceedings

of

Conference

on

Empirical

Methods

in

Natural

Language

Processing

-LSB-

-RSB-

Cortes

C.

and

Vapnik

V.

Support-vector

networks

Machine

Learning

-LSB-

-RSB-

Chieu

H.

L.

and

Ng

H.

T

A

maximum

entropy

approach

to

information

extraction

from

semi-structured

and

free

text

In

Proceedings

of

the

Eighteenth

National

Conference

on

Artificial

Intelligence

-LSB-

-RSB-

Evans

D.

K.

Klavans

J.

L.

and

McKeown

K.

R.

Columbia

newsblaster

multilingual

news

summarization

on

the

Web

In

Proceedings

of

Human

Language

Technology

conference

North

American

chapter

of

the

Association

for

Computational

Linguistics

annual

meeting

-LSB-

-RSB-

Ghahramani

Z.

and

Jordan

M.

I.

Factorial

hidden

markov

models

Machine

Learning

-LSB-

-RSB-

Gheel

J.

and

Anderson

T.

Data

and

metadata

for

finding

and

reminding

In

Proceedings

of

the

International

Conference

on

Information

Visualization

-LSB-

-RSB-

Giles

C.

L.

Petinot

Y.

Teregowda

P.

B.

Han

H.

Lawrence

S.

Rangaswamy

A.

and

Pal

N.

eBizSearch

a

niche

search

engine

for

e-Business

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

Giuffrida

G.

Shek

E.

C.

and

Yang

J.

Knowledge-based

metadata

extraction

from

PostScript

files

In

Proceedings

of

the

Fifth

ACM

Conference

on

Digital

Libraries

-LSB-

-RSB-

Han

H.

Giles

C.

L.

Manavoglu

E.

Zha

H.

Zhang

Z.

and

Fox

E.

A.

Automatic

document

metadata

extraction

using

support

vector

machines

In

Proceedings

of

the

Third

ACM/IEEE-CS

Joint

Conference

on

Digital

Libraries

-LSB-

-RSB-

Kobayashi

M.

and

Takeda

K.

Information

retrieval

on

the

Web

ACM

Computing

Surveys

-LSB-

-RSB-

Lafferty

J.

McCallum

A.

and

Pereira

F.

Conditional

random

fields

probabilistic

models

for

segmenting

and

labeling

sequence

data

In

Proceedings

of

the

Eighteenth

International

Conference

on

Machine

Learning

-LSB-

-RSB-

Li

Y.

Zaragoza

H.

Herbrich

R.

Shawe-Taylor

J.

and

Kandola

J.

S

The

perceptron

algorithm

with

uneven

margins

In

Proceedings

of

the

Nineteenth

International

Conference

on

Machine

Learning

-LSB-

-RSB-

Liddy

E.

D.

Sutton

S.

Allen

E.

Harwell

S.

Corieri

S.

Yilmazel

O.

Ozgencil

N.

E.

Diekema

A.

McCracken

N.

and

Silverstein

J.

Automatic

Metadata

generation

evaluation

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

Littlefield

A.

Effective

enterprise

information

retrieval

across

new

content

formats

In

Proceedings

of

the

Seventh

Search

Engine

Conference

-LSB-

-RSB-

Mao

S.

Kim

J.

W.

and

Thoma

G.

R

A

dynamic

feature

generation

system

for

automated

metadata

extraction

in

preservation

of

digital

materials

In

Proceedings

of

the

First

International

Workshop

on

Document

Image

Analysis

for

Libraries

-LSB-

-RSB-

McCallum

A.

Freitag

D.

and

Pereira

F.

Maximum

entropy

markov

models

for

information

extraction

and

segmentation

In

Proceedings

of

the

Seventeenth

International

Conference

on

Machine

Learning

-LSB-

-RSB-

Murphy

L.

D.

Digital

document

metadata

in

organizations

roles

analytical

approaches

and

future

research

directions

In

Proceedings

of

the

Thirty-First

Annual

Hawaii

International

Conference

on

System

Sciences

-LSB-

-RSB-

Pinto

D.

McCallum

A.

Wei

X.

and

Croft

W.

B.

Table

extraction

using

conditional

random

fields

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

Ratnaparkhi

A.

Unsupervised

statistical

models

for

prepositional

phrase

attachment

In

Proceedings

of

the

Seventeenth

International

Conference

on

Computational

Linguistics

-LSB-

-RSB-

Robertson

S.

Zaragoza

H.

and

Taylor

M.

Simple

extension

to

multiple

weighted

fields

In

Proceedings

of

ACM

Thirteenth

Conference

on

Information

and

Knowledge

Management

-LSB-

-RSB-

Yi

J.

and

Sundaresan

N.

Metadata

based

Web

mining

for

relevance

In

Proceedings

of

the

International

Symposium

on

Database

Engineering

Applications

-LSB-

-RSB-

Yilmazel

O.

Finneran

C.

M.

and

Liddy

E.

D.

MetaExtract

An

NLP

system

to

automatically

assign

metadata

In

Proceedings

of

the

Joint

ACM/IEEE

Conference

on

Digital

Libraries

-LSB-

-RSB-

Zhang

J.

and

Dimitroff

A.

Internet

search

engines

response

to

metadata

Dublin

Core

implementation

Journal

of

Information

Science

-LSB-

-RSB-

Zhang

L.

Pan

Y.

and

Zhang

T.

Recognising

and

using

named

entities

focused

named

entity

recognition

using

machine

learning

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

O B O O O O O O B I O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O

search
powerpoint documents
metadata extraction
machine learning
file formats
extraction

generic languages
microsoft office automation
information search and retrieval
file extraction
digital copies
metadata processing
search ranking results
precision extraction
information extraction


An Index System For Quality Synthesis Evaluation Of Btoc Business Website It is important for successful electronic business to have a hi-quality business website . So we need an accurate and effective index system to evaluate and analyses the quality of the business website . In this paper , the evaluation index system following the ` grey box ' principle is proposed which considers both efficiency of business website and performance of electronic business system . Using R-Hierarchical clustering method to extract the typical indexes from sub-indexes is theoretically proved to have a rationality and effectiveness . Finally , the evaluation method is briefly discussed . TP393 .4 Business website is an online media between buyer and seller . A hi-quality website is crucial to a company for a successful e-business . What is a hi-quality business website ? In terms of maintaining the website , what do we focus on so that the quality meets the users ' needs ? Apparently , using click-through rate to assess the popularity can not objectively and accurately evaluate the quality of the business websites . Instead , we need to rely on scientific evaluation index system and methods . At present , there are many methods available for business website comparison or ranking , such as Usage Ranking , Purchase Comparison , Expert Opinion and Synthesis Evaluation etc. . You can find both official authority and non-governmental organization that issue their power ranking . The former one is to monitor and regulate the market , such as CNNIC , which organized the competition for the Top Ten Websites in domestic . The latter one , such as Consumerreports -LRB- www.consumerreports . org -RRB- , BizRate -LRB- www.bizrate.com -RRB- , Forrester Research etc. , is mainly to guide the web users ' activity . These kinds of comparison or ranking have special value in getting reputation and increasing recognition of the business websites among the users , however , e-business enterprise can not improve the quality of their websites directly based on the results of these kinds of assessments . The main purpose of this paper is to develop an index system for quantitative evaluation of the BtoC websites , which dose not emphasize the income of the website but focus on evaluating of its synthesis quality . We hope that the applying of this index system will provide the technique developers and maintainers some references for designing , appraising and diagnosing their e-business system to improve its quality level , and to support managers to make decisions for operation of the websites . Comparing to the fast growing of e-business websites in the world , currently we can rarely find the particular research on the evaluation index system of business website . QEM -LRB- The website quality evaluation method -RRB- proposed by Olsina and Godoy etc. in 1999 can be considered as one of the representative approaches . It based on the main factors to evaluate the quality of the websites , including functionality -LRB- global search , navigability , and content relativity -RRB- , usability -LRB- website map , addresses directory -RRB- , efficiency and reliability . In 2000 , American researcher , Panla Solaman , presented e-SERVQUAL model based on the conventional service quality evaluation model SERVQUAL . It contains some factors like efficiency , deal completeness , reliability , privacy protection , responsiveness , recompense and contact etc. . In the same year , another American researcher , Hauler , introduced an e-QUAL model which includes the factors of content , accessibility , navigability , design and presentation , responsiveness , background , personalization and customization , etc. . In 2004 , F.J. Miranda Gonzalez and T.M.Banegil Palacios developed an universal evaluation index system WIS -LRB- Web Assessment Index -RRB- that can be employed to assess websites by different organizations . It consists of four indexes of accessibility , navigability , speed and content . -LSB- 1 -RSB- However , the universal index system can not measure a website exactly and absolutely due to the industry specialty , organizational characteristics and different usages . One of the representative researches is Mr. ZhongHai Li 's paper about ergonomics standard of online store . It assesses the business websites by testing if the design of the website coincides with the shopping process of online consumers . This standard has five factors , such as search and browse , merchandise information , shopping cart , register and pay , service and support . -LSB- 4 -RSB- Another index system for small and medium business websites covers the factors of general features , design , promotion , information and the others . -LSB- 5 -RSB- Here we list our major findings from the previous researches : 2.1 Unreasonable Selection of the Index Some research consider not only the original design but also the factors such as promotion and income of business website . 75 Some evaluation systems have correlative or contradictive indexes . For example , it considers the download speed , at the same time , it requires the web designers not to excessively use flash and sounds to slow down the speed . 2.2 Unilateral Evaluation Most of the research takes the users ' view to evaluate the function and design of website . It treats the business system as a ` black box ' and ignores the impact of system performance on the websites quality . But considering the factors of system performance alone is also not a complete evaluation for improving service quality of website . 2.3 Lack of a Complete Set of Quality Synthesis Evaluation System A complete set of tool to evaluate the websites must include the following important elements : categories , factors , weights , rankings standard and assessment model . So far , we have not seen any literature discussing complete set of evaluation index system aiming at the quality of BtoC websites . First , the three fundamental principles we need to follow are to be comprehensive , to be scientific and to be feasible . We should evaluate all the facets of the website from different dimensions and avoid missing value of important factors . Moreover , the definition of the evaluation index should be accurateobjective and logical so it can eliminate the impact on the evaluation result brought by the correlative indexes . Concurrently , we need reduce the quantity of indexes or adopt the simple ones which data is easier to be collected , and prevent from complicated calculation due to the excessive indexes . The main purpose of improving business websites is to serve the users better . They are concerned only about the websites ' external attributes , such as content , function , presentation and browse speed , etc. . So , evaluating only by taking their views can not directly guide to develop , maintain and administrate the website . Just like treating the patient 's symptom but not the disease itself , the technique developer or maintainer can not radically improve the quality of their websites by correcting system structure and web design according to the evaluation result . Only after we adopt the ` grey box ' index system that considers both efficiency of business website and performance of e-business system , we can establish a quality synthesis evaluation index system to benefit the management of BtoC websites . Selection of index items lays down the foundation for constructing evaluation index system . After we thoroughly analyze the evaluation objectives based on the characteristics of business website , we propose an initial index system includes 5 categories and totally 28 index items shown in the following Table 1 . Table 1 Quality evaluation indexes for business websites Categories Indexes Function Effectiveness 1 Integrative Function 2 Interactive Function 3 Convenience 4 Service Personalization 5 Website Credibility 6 Business Authorization Business Information 7 Accuracy 8 Authoritativeness 9 Variety in Type 10 Inclusiveness 11 Uniqueness 12 Orderliness 13 Timeliness 14 Variety in Search Method 15 Search Effectiveness 16 Version Internationalization Website Design 17 User Interface Friendliness 18 Development Standardization 19 Website Uniqueness 20 Columns Originality 21 Website Structure Clarity 22 Page Style Consistency 23 Harmonization System Usability 24 System Stableness 25 Compatibility 26 System Security 27 Self-adaptability System Efficiency 28 System Speediness Website self-adaptability refers to capability of e-business system intelligently providing personalized service and dynamic optimizing system performance . System Efficiency refers to the ability that the system response quickly to the requests of numbers of web users . It can be measured through values of some quantitative indexes , such as response time , throughput or utilization rate , etc. . It is necessary for our initial evaluation system to optimize if it can be applied in practice . First , the indexes are more or less correlative which will affects the objectiveness of the evaluation . Second , there are too more indexes that will result in lower efficiency . Therefore , we try to extract and simplify the indexes by using R-Hierarchical clustering method . Generally , R indicates the coefficient of correlation between two items . R-Hierarchical clustering method is usually applied to cluster the indexes . The steps are described as following . 5.1 Calculate Coefficient of Correlation and Clustering It firstly treats every index as one cluster . So , we have 28 clusters . Then , coefficient of correlation is calculated between every two clusters by minimum-distance method . Next , the two clusters with the maximal coefficient of correlation are clustered into a new one . The same process is repeated until all the indexes are clustered into one . 76 5.2 Analyze the Clustering Process and Determine Clusters We analyze the variation of minimum coefficient of correlation during the clustering process to find the leap points . According to the number of leap points and the knowledge of special field , we can eventually determine how many clusters we need . The whole process is illustrated in the following Figure 1 . Figure 1 The process of R-Hierarchical clustering Following the principle of simplification and feasibility and considering the characteristics of BtoC website , we cluster the 28 index items into 10 . The precision rate is over 0.75 . 5.3 Calculate Correlation Index and Extract the Representative Indexes First , we calculate the correlation index that is the average of R between one index and every other index in the same cluster . 1 2 2 = j j m r R mi in this formula is the number of the indexes in the cluster that index Xj belongs to . Then , we select the index with the maximal correlation index in the total 10 clusters individually and identify 10 of them as the most representative indexes . Finally , the weights of the indexes are derived by the expert grade method . The final indexes and their weights are shown in the following table 2 . Table 2 The final indexes and their weights Category Weight Index Weight 1.1 Service Personalization 0.10 Function Effectiveness 0.22 1.2 Website Credibility 0.12 2.1 Information Inclusiveness 0.10 Business Information 0.18 2.2 Version Internationalization 0.08 3.1 Columns Originality 0.09 Website Design 0.28 3.2 Website Structure Clarity 0.10 3.3 Harmonization 0.09 4.1 System Stableness 0.10 System Usability 0.22 4.2 System Security 0.12 System Efficiency 0.10 5.1 System Speediness 0.10 In this paper , we have proposed an index system for quality synthesis evaluation and diagnosis of the BtoC websites following the ` Grey Box ' evaluation principle , and scientifically determined and simplified the index items . Usually , factor analysis or principal component analysis is used to solve the problem of common-factor and multiple indexes . But these methods are only suitable for the quantitative indexes , and the evaluation process is not truly simplified . Because the new index is the linear function of some original ones , it still needs to calculate the value of new indexes by collecting all the values of the original ones . In our index system , most of index is descriptive one . So we have finalized the indexes by using the R-Hierarchical clustering method . It really has reduced the number of the evaluation indexes without losing the major information from the original indexes . Furthermore , it has effectively avoided the impact of common-factors on the evaluation result . Only the index of system efficiency can be measured through quantitative sub-indexes such as response time , etc. . Most of depictive indexes are subjective and fuzzy . In view of this , we should use fuzzy comprehensive analysis method to evaluate to get more efficiency result . In our future work we are intended to propose an evaluation model and conduct evaluation to some famous domestic BtoC websites to prove if this index system is scientific and feasible . Moreover , we will improve this set of index system including evaluation model to make the whole set of index system more feasibility . -LSB- 1 -RSB- F.J. Miranda Gonzalez , T.M. Banegil Palacios , Quantitative evaluation of commercial web sites : an empirical study of Spanish firms , International Journal of Information Management , 24 -LRB- 2004 -RRB- 313-328 -LSB- 2 -RSB- Chang Liu , Kirk P. Amett , Exploring the factors associated with Web site success in the context of electronic commerce , Information & Management , 38 -LRB- 2000 -RRB- 23-33 -LSB- 3 -RSB- Evans , J. R. , & King , V. E. . Business-to-business marketing and the World Wide Web : Planning , managing and assessing web sites . Industrial Marketing Management , 28 -LRB- 1999 -RRB- 343 358 -LSB- 4 -RSB- Zhonghai Li , Jianqiao Liao , Hui Xiao ; the analyze of work efficiency on webshop design in China ; human work efficiency ; 4 -LRB- 2002 -RRB- 43-45 -LSB- 5 -RSB- Research a index system for evaluation of on enterprise website , http://www.365un.com/ xmb/viewthread.php ? tid = 1 998 77

O O O O B I I O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

clustering
business website
quality synthesis evaluation
system performance
evaluation system
correlation index

b2c websites
quality evaluation index
r-hierarchical clustering method
representitive indexes
fuzzy analysis
index optimization
r-hierarchical clustering
e-commerce