Focused

Named

Entity

Recognition

Using

Machine

Learning

In

this

paper

we

study

the

problem

of

finding

most

topical

named

entities

among

all

entities

in

a

document

which

we

refer

to

as

focused

named

entity

recognition

We

show

that

these

focused

named

entities

are

useful

for

many

natural

language

processing

applications

such

as

document

summarization

search

result

ranking

and

entity

detection

and

tracking

We

propose

a

statistical

model

for

focused

named

entity

recognition

by

converting

it

into

a

classification

problem

We

then

study

the

impact

of

various

linguistic

features

and

compare

a

number

of

classification

algorithms

From

experiments

on

an

annotated

Chinese

news

corpus

we

demonstrate

that

the

proposed

method

can

achieve

near

human-level

accuracy

I.

-LSB-

Artificial

Intelligence

-RSB-

Natural

language

Processing

Text

Analysis

H.

-LSB-

Information

Storage

And

Retrieval

-RSB-

Content

Analysis

and

Indexing

Linguistic

processing

Algorithms

Experimentation

With

the

rapid

growth

of

online

electronic

documents

many

technologies

have

been

developed

to

deal

with

the

enormous

amount

of

information

such

as

automatic

summarization

topic

detection

and

tracking

and

information

retrieval

Among

these

technologies

a

key

component

is

to

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

SIGIR

July

Sheffield

South

Yorkshire

UK

Copyright

ACM

identify

the

main

topics

of

a

document

where

topics

can

be

represented

by

words

sentences

concepts

and

named

entities

A

number

of

techniques

for

this

purpose

have

been

proposed

in

the

literature

including

methods

based

on

position

-LSB-

-RSB-

cue

phrases

-LSB-

-RSB-

word

frequency

lexical

chains

-LSB-

-RSB-

and

discourse

segmentation

-LSB-

-RSB-

Although

word

frequency

is

the

easiest

way

to

representing

the

topics

of

a

document

it

was

reported

in

-LSB-

-RSB-

that

position

methods

produce

better

results

than

word

counting

based

methods

Important

sentence

extraction

is

the

most

popular

method

studied

in

the

literature

A

recent

trend

in

topic

sentence

extraction

is

to

employ

machine

learning

methods

For

example

trainable

classifiers

have

been

used

in

-LSB-

-RSB-

to

select

sentences

based

on

features

such

as

cue

phrase

location

sentence

length

word

frequency

and

title

etc.

All

of

the

above

methods

share

the

same

goal

of

extracting

important

sentences

from

documents

However

for

topic

representation

sentence-level

document

summaries

may

still

contain

redundant

information

For

this

reason

other

representations

have

also

been

suggested

For

example

in

-LSB-

-RSB-

the

authors

used

structural

features

of

technical

papers

to

identify

important

concepts

rather

than

sentences

The

authors

of

-LSB-

-RSB-

presented

an

efficient

algorithm

to

choose

topic

terms

for

hierarchical

summarization

according

to

a

proba-bilistic

language

model

Another

hybrid

system

presented

in

-LSB-

-RSB-

generated

summarizations

with

the

help

of

named

entity

foci

of

an

article

These

named

entities

include

people

organizations

and

places

and

untyped

names

In

this

paper

we

study

the

problem

of

finding

important

named

entities

from

news

articles

which

we

call

focused

named

entity

recognition

A

news

article

often

reports

an

event

that

can

be

effectively

summarized

by

the

five

W

-LRB-

who

what

when

where

and

why

-RRB-

approach

Many

of

the

five

W

's

can

be

associated

with

appropriate

named

entities

in

the

article

Our

definition

of

focused

named

entities

is

mainly

concerned

with

Who

and

What

Therefore

it

is

almost

self-evident

that

the

concept

of

focused

named

entity

is

important

for

document

understanding

and

automatic

information

extraction

In

fact

a

number

of

recent

studies

have

already

suggested

that

named

entities

are

useful

for

text

summarization

-LSB-

-RSB-

Moreover

we

shall

illustrate

that

focused

named

entities

can

be

used

in

other

text

processing

tasks

as

well

For

example

we

can

rank

search

results

by

giving

more

weights

to

focused

named

entities

We

define

focused

named

entities

as

named

entities

that

are

most

relevant

to

the

main

topic

of

a

news

article

Our

task

is

to

automatically

select

these

focused

named

entities

from

the

set

of

all

entities

in

a

document

Since

focused

named

entity

recognition

is

a

newly

proposed

machine

learning

task

we

need

to

determine

whether

it

is

well-posed

That

is

whether

there

exists

a

sufficient

level

of

agreement

on

focused

named

entities

among

human

reviewers

A

detailed

study

on

this

matter

will

be

reported

in

the

section

The

conclusion

of

our

study

is

that

there

is

indeed

a

sufficient

level

of

agreement

Encouraged

by

this

study

we

further

investigated

the

machine

learning

approach

to

this

problem

which

is

the

focus

of

the

paper

We

discuss

various

issues

encountered

in

the

process

of

building

a

machine

learning

based

system

and

show

that

our

method

can

achieve

near

human

performance

The

remainder

of

this

paper

is

organized

as

follows

In

Section

we

introduce

the

problem

of

focused

named

entity

recognition

and

illustrate

its

applications

Section

describes

a

general

machine

learning

approach

to

this

problem

In

Section

we

present

features

used

in

our

system

Section

presents

a

study

of

human-level

agreement

on

focused

named

entities

and

various

experiments

which

illustrate

the

importance

of

different

features

Some

final

conclusions

will

be

given

in

section

Figure

is

an

example

document

This

article

reports

that

Boeing

Company

would

work

with

its

new

Research

and

Technology

Center

to

develop

a

new

style

of

electric

airplane

On

the

upper

half

of

the

page

we

list

all

named

entities

appearing

in

the

article

and

mark

the

focused

entities

Among

the

twelve

named

entities

Boeing

Company

and

its

Research

and

Technology

Center

are

most

relevant

to

the

main

topic

Here

we

call

Boeing

Company

and

Research

and

Technology

Center

the

focuses

Clearly

focused

named

entities

are

important

for

representing

the

main

topic

of

the

content

In

the

following

we

show

that

the

concept

of

focused

named

entity

is

useful

for

many

natural

language

processing

applications

such

as

summarization

search

ranking

and

topic

detection

and

tracking

Using

Focused

Named

Entity

for

Summarization

We

consider

the

task

of

automatic

summarization

of

the

sample

document

in

Figure

A

traditional

method

is

to

select

sentences

with

highest

weights

where

sentence

weights

are

calculated

by

averaging

term

frequencies

of

words

it

contains

The

resulting

summarization

is

given

in

Figure

Using

focused

named

entities

we

consider

two

methods

to

refine

the

above

summarization

The

first

method

is

to

increase

the

weight

of

the

focused

named

entity

Boeing

in

the

sentences

leading

to

the

summary

in

Figure

The

other

method

simply

picks

sentences

containing

the

focused

named

entity

Boeing

as

in

Figure

From

this

example

we

can

see

that

summarization

using

focused

named

entities

gives

more

indicative

description

of

an

article

Using

Focused

Named

Entity

for

Ranking

Search

Results

Suppose

we

want

to

find

news

about

World

Cup

football

match

from

a

collection

of

news

articles

First

we

search

The

original

article

can

be

accessed

at

Figure

Sample

document

with

focused

named

entities

marked

Boeing

To

Explore

Electric

Airplane

Fuel

cells

and

electric

motors

will

not

replace

jet

engines

on

commercial

transports

but

they

could

one

day

replace

gas

turbine

auxiliary

power

units

Unlike

a

battery

which

needs

to

be

recharged

fuel

cells

keep

working

as

long

as

the

fuel

lasts

Fuel

cells

show

the

promise

of

one

day

providing

efficient

essentially

pollution-free

electrical

power

for

commercial

airplane

primary

electrical

power

needs

Daggett

said

Figure

Summary

using

term

frequency

weighting

Boeing

To

Explore

Electric

Airplane

Boeing

Commercial

Airplanes

will

develop

and

test

an

electrically

powered

demonstrator

airplane

as

part

of

a

study

to

evaluate

environmentally

friendly

fuel

cell

technology

for

future

Boeing

products

Fuel

cells

and

electric

motors

will

not

replace

jet

engines

on

commercial

transports

but

they

could

one

day

replace

gas

turbine

auxiliary

power

units

By

adapting

this

technology

for

aviation

Boeing

intends

to

demonstrate

its

leadership

in

the

pursuit

of

delivering

environmentally

preferred

products

Figure

Summary

weighted

by

focused

named

entities

Boeing

To

Explore

Electric

Airplane

Boeing

Commercial

Airplanes

will

develop

and

test

an

electrically

powered

demonstrator

airplane

as

part

of

a

study

to

evaluate

environmentally

friendly

fuel

cell

technology

for

future

Boeing

products

The

airplane

manufacturer

is

working

with

Boe-ing

's

new

Research

and

Technology

Center

in

Madrid

Spain

to

modify

a

small

single-engine

airplane

by

replacing

its

engine

with

fuel

cells

and

an

electric

motor

that

will

turn

a

conventional

propeller

Boeing

Madrid

will

design

and

integrate

the

experimental

airplane

's

control

system

By

adapting

this

technology

for

aviation

Boeing

intends

to

demonstrate

its

leadership

in

the

pursuit

of

delivering

environmentally

preferred

products

Figure

Summary

using

sentences

containing

focused

named

entities

for

documents

containing

the

key

phrase

World

Cup

The

ranking

function

which

determines

which

document

is

more

relevant

to

the

query

is

very

important

to

the

search

quality

Since

our

query

is

a

single

phrase

the

ranked

search

results

displayed

in

Table

are

based

on

the

term

frequency

of

the

phrase

World

Cup

It

is

clear

that

without

deeper

text

understanding

term

frequency

is

a

quite

reasonable

measure

of

relevancy

However

although

some

articles

may

contain

more

World

Cup

than

others

they

may

actually

focus

less

on

the

World

Cup

event

which

we

are

interested

Therefore

a

better

indication

of

document

relevancy

is

whether

a

document

focuses

on

the

entity

we

are

interested

in

A

simple

method

is

to

re-order

the

search

results

first

by

whether

the

query

entity

is

focused

or

not

and

then

by

its

term-frequency

It

is

quite

clear

that

this

method

gives

higher

quality

ranking

In

this

example

we

use

Chinese

corpus

for

demonstration

so

the

original

searching

results

are

in

Chinese

which

we

have

translated

into

English

for

reading

convenience

Other

Uses

of

Focused

Named

Entity

We

believe

that

focused

named

entities

are

also

helpful

in

text

clustering

and

categorization

tasks

such

as

topic

detection

and

tracking

This

is

because

if

focused

named

entities

are

automatically

recognized

then

the

event

for

each

document

can

be

described

more

precisely

Since

focused

named

entities

characterize

what

an

article

talks

about

it

is

natural

to

organize

articles

based

on

them

Therefore

by

giving

more

weights

to

focused

named

entities

we

believe

that

we

can

potentially

obtain

better

quality

clustering

and

more

accurate

topic

detection

and

tracking

Our

study

of

the

focused

named

entity

recognition

problem

is

motivated

by

its

potential

applications

as

illustrated

above

Experiments

in

section

indicate

that

there

is

a

sufficient

agreement

on

focused

named

entities

among

human

reviewers

Therefore

our

goal

is

to

build

a

system

that

can

automatically

detect

focused

named

entities

among

all

named

entities

in

a

document

We

shall

mention

that

although

this

paper

only

studies

named

entities

the

basic

idea

can

be

extended

to

tasks

such

as

finding

important

words

noun-phrases

in

a

document

Focused

named

entity

recognition

can

be

regarded

as

a

binary

classification

problem

Consider

the

set

of

all

named

entities

in

a

document

extracted

by

a

named

entity

recognition

system

Each

entity

in

this

set

can

be

labeled

yes

if

it

is

a

focused

entity

or

no

if

it

is

not

We

formally

define

a

two-class

categorization

problem

as

one

to

determine

a

label

y

-LCB-

-RCB-

associated

with

a

vector

x

of

input

variables

However

in

order

to

build

a

successful

focused

named

entity

extractor

a

number

of

issues

have

to

be

studied

First

named

entities

that

refer

to

the

same

person

or

organization

need

to

be

grouped

together

secondly

what

features

are

useful

and

thirdly

how

well

different

learning

algorithms

perform

on

this

task

These

issues

will

be

carefully

studied

Coreference

Resolution

Coreference

is

a

common

phenomenon

in

natural

language

It

means

that

an

entity

can

be

referred

to

in

different

ways

and

in

different

locations

of

the

text

Therefore

for

focused

named

entity

recognition

it

is

useful

to

apply

a

coreference

resolution

algorithm

to

merge

entities

with

the

same

referents

in

a

given

document

There

are

different

kinds

of

coreference

according

to

the

basic

coreference

types

such

as

pronominal

coreference

proper

name

coreference

apposition

predicate

nominal

etc.

Here

in

our

system

we

only

consider

proper

name

coreference

which

is

to

identify

all

variations

of

a

named

entity

in

the

text

Although

it

is

possible

to

use

machine

learning

methods

for

coreference

resolution

-LRB-

see

-LSB-

-RSB-

as

an

example

-RRB-

we

shall

use

a

simpler

scheme

which

works

reasonably

well

Our

coreference

resolution

method

can

be

described

as

follows

Partitioning

The

set

of

named

entities

is

divided

into

sub-sets

according

to

named

entity

types

because

coreference

only

occurs

among

entities

with

the

same

types

Pair-wise

comparison

Within

each

sub-set

pair-wise

comparison

is

performed

to

detect

whether

each

entity-pair

is

an

instance

of

coreference

In

this

study

we

use

a

simple

algorithm

which

is

based

on

string-matching

only

Since

we

work

with

Chinese

data

we

split

each

entity

into

single

Chinese

characters

We

study

two

different

schemes

here

using

either

exact

string

matching

or

partial

string

matching

to

decide

coreference

In

the

case

of

exact

string

matching

two

entities

are

considered

to

be

a

coreference

pair

only

when

they

are

identical

In

the

case

of

partial

string

matching

if

characters

in

the

shorter

entity

form

a

-LRB-

non-consecutive

-RRB-

sub-string

of

the

longer

entity

then

the

two

entities

are

considered

to

be

a

coreference

pair

Clustering

Merge

all

coreference

pairs

created

in

the

second

step

into

the

same

coreference

chains

This

step

can

also

be

done

differently

For

example

by

using

a

sequential

clustering

method

Although

the

coreference

resolution

algorithm

described

above

is

not

perfect

it

is

not

crucial

since

the

results

will

Table

Search

result

of

World

Cup

focus/not

tf

title

focus

Uncover

the

Mystery

of

World

Cup

Draws

focus

Brazil

and

Germany

Qualified

Iran

Kicked

out

focus

Preparing

for

World

Cup

China

Football

Federation

and

Milutinovic

Snatch

the

Time

focus

Sun

Wen

Understands

the

Pressure

Milutinovic

and

China

Team

Faced

focus

Korea

Leaves

More

Tickets

to

China

Fans

focus

Paraguay

Qualified

but

Head

Coach

Dismissed

no

LiXiang

Special

Relationships

between

Milutinovic

and

I

no

Three

Stars

on

Golden

Eagle

Festival

focus

Adidas

Fevernova

the

Official

FIFA

World

Cup

Ball

Appears

Before

the

Public

in

Beijing

no

China

's

World

Top

Start

to

Vote

focus

Qualified

vs.

Kicked

out

McCarthy

Stays

on

Blazevic

Demits

focus

China

Attends

Group

Match

in

Korea

But

not

in

the

Same

Group

With

Korea

no

Do

n't

Scare

Peoples

with

Entering

WTO

no

Kelon

Tops

China

's

Home

Appliance

Industry

in

CCTV

Ads

Bidding

no

Lou

Lan

Great

Secrets

Behind

focus

Australia

Beats

Uruguay

by

One

Goal

no

Chang

Hong

's

King

of

Precision

Display

Good

Friends

of

Football

Fans

be

passed

to

a

machine

learning

algorithm

in

a

later

stage

which

can

offset

the

mistakes

made

in

the

coreference

stage

Our

experiment

shows

that

by

using

coreference

resolution

the

overall

system

performance

can

be

improved

appreciably

Classification

Methods

In

this

paper

we

compare

three

methods

a

decision

tree

based

rule

induction

system

a

naive

Bayes

classifier

and

a

regularized

linear

classification

method

based

on

robust

risk

minimization

Decision

Tree

In

text-mining

application

model

interpretability

is

an

important

characteristic

to

be

considered

in

addition

to

the

accuracy

achieved

and

the

computational

cost

The

requirement

of

interpretability

can

be

satisfied

by

using

a

rule-based

system

such

as

rules

obtained

from

a

decision

tree

Rule-based

systems

are

particularly

appealing

since

a

person

can

examine

the

rules

and

modify

them

It

is

also

much

easier

to

understand

what

a

system

does

by

examining

its

rules

We

shall

thus

include

a

decision

tree

based

classifier

in

this

study

In

a

typical

decision

tree

training

algorithm

there

are

usually

two

stages

The

first

stage

is

tree

growing

where

a

tree

is

built

by

greedily

splitting

each

tree

node

based

on

a

certain

figure

of

merit

However

after

the

first

stage

the

tree

can

overfit

the

training

data

therefore

a

second

stage

involving

tree

pruning

is

invoked

In

this

stage

one

removes

overfitted

branches

of

the

tree

so

that

the

remaining

portion

has

better

predictive

power

In

our

decision

tree

package

the

splitting

criteria

during

tree

growth

is

similar

to

that

of

the

standard

program

-LSB-

-RSB-

and

the

tree

pruning

is

done

using

a

Bayesian

model

combination

approach

See

-LSB-

-RSB-

for

detailed

description

Naive

Bayes

Another

very

popular

binary

classification

method

is

naive

Bayes

In

spite

of

its

simplicity

it

often

achieves

reasonable

performance

in

practical

applications

It

can

be

regarded

as

a

linear

classification

method

where

we

seek

a

weight

vector

w

and

a

threshold

such

that

w

T

x

if

its

label

y

and

w

T

x

if

its

label

y

A

score

of

value

w

T

x

can

be

assigned

to

each

data

point

as

a

surrogate

for

the

likelihood

of

x

to

be

in

class

In

this

work

we

adopt

the

multinomial

model

described

in

-LSB-

-RSB-

Let

-LCB-

-LRB-

x

y

-RRB-

-LRB-

x

n

y

n

-RRB-

-RCB-

be

the

set

of

training

data

The

linear

weight

w

is

given

by

w

w

w

and

Denote

by

x

i

j

the

j-th

component

of

the

data

vector

x

i

then

the

j-th

component

w

c

j

of

w

c

-LRB-

c

-RRB-

is

given

by

w

c

j

log

i

y

i

c

x

i

j

d

d

j

i

y

i

c

x

i

j

and

c

-LRB-

c

-RRB-

is

given

by

c

log

-LCB-

i

y

i

c

-RCB-

n

The

parameter

in

the

above

formulation

is

a

smoothing

-LRB-

regularization

-RRB-

parameter

-LSB-

-RSB-

fixed

to

be

which

corresponds

to

the

Laplacian

smoothing

Robust

Risk

Minimization

Method

Similar

to

naive

Bayes

this

method

is

also

a

linear

prediction

method

Given

a

linear

model

p

-LRB-

x

-RRB-

w

T

x

b

we

consider

the

following

prediction

rule

predict

y

if

p

-LRB-

x

-RRB-

and

predict

y

otherwise

The

classification

error

-LRB-

we

shall

ignore

the

point

p

-LRB-

x

-RRB-

which

is

assumed

to

occur

rarely

-RRB-

is

I

-LRB-

p

-LRB-

x

-RRB-

y

-RRB-

if

p

-LRB-

x

-RRB-

y

if

p

-LRB-

x

-RRB-

y

A

very

natural

way

to

compute

a

linear

classifier

is

by

finding

a

weight

-LRB-

w

b

-RRB-

that

minimizes

the

average

classification

error

in

the

training

set

-LRB-

w

b

-RRB-

arg

min

w

b

n

n

i

I

-LRB-

w

T

x

i

b

y

i

-RRB-

Unfortunately

this

problem

is

typically

NP-hard

computa-tionally

It

is

thus

desirable

to

replace

the

classification

error

loss

I

-LRB-

p

y

-RRB-

with

another

formulation

that

is

computation-ally

more

desirable

Large

margin

methods

such

as

SVM

employ

modified

loss

functions

that

are

convex

Many

loss

functions

work

well

for

related

classification

problems

such

as

text-categorization

-LSB-

-RSB-

The

specific

loss

function

consider

here

is

h

-LRB-

p

y

-RRB-

py

py

-LRB-

py

-RRB-

py

-LSB-

-RSB-

py

That

is

our

linear

weights

are

computed

by

minimizing

the

following

average

loss

on

the

training

data

-LRB-

w

b

-RRB-

arg

min

w

n

n

i

h

-LRB-

w

T

x

i

b

y

i

-RRB-

This

method

which

we

refer

to

as

RRM

-LRB-

robust

risk

minimization

-RRB-

has

been

applied

to

linguistic

processing

-LSB-

-RSB-

and

text

categorization

-LSB-

-RSB-

with

good

results

Detailed

algorithm

was

introduced

in

-LSB-

-RSB-

We

assume

that

named

entities

are

extracted

by

a

named

entity

recognition

system

Many

named

entity

recognition

techniques

have

been

reported

in

the

literal

most

of

them

use

machine

learning

approach

An

overview

of

these

methods

can

be

found

in

-LSB-

-RSB-

In

our

system

for

the

purpose

of

simplicity

we

use

human

annotated

named

entities

in

the

experiments

In

the

learning

phase

each

named

entity

is

considered

as

an

independent

learning

instance

Features

must

reflect

properties

of

an

individual

named

entity

such

as

its

type

and

frequency

and

various

global

statistical

measures

either

at

the

document

scale

or

at

the

corpus

scale

This

section

describes

features

we

have

considered

in

our

system

our

motivations

and

how

their

values

are

encoded

Entity

Type

Four

entity

types

are

defined

person

organization

place

and

proper

nouns

The

type

of

a

named

entity

is

a

very

useful

feature

For

example

person

and

organization

are

more

likely

to

be

the

focus

than

a

place

Each

entity

type

corresponds

to

a

binary

feature-component

in

the

feature

vector

taking

a

value

of

either

one

or

zero

For

example

a

person

type

is

encoded

as

-LSB-

-RSB-

and

an

organization

type

is

encoded

as

-LSB-

-RSB-

In

Title

or

Not

Whether

a

named

entity

appears

in

the

title

or

not

is

an

important

indicator

of

whether

it

is

a

focused

entity

This

is

because

title

is

a

concise

summary

of

what

an

article

is

about

The

value

of

this

feature

is

binary

-LRB-

or

-RRB-

Entity

Frequency

This

feature

is

the

number

of

times

that

the

named

entity

occurs

in

the

document

Generally

speaking

the

more

frequent

it

occurs

the

more

important

it

is

The

value

of

this

feature

is

just

the

frequency

of

the

named

entity

Entity

Distribution

This

feature

is

somewhat

complicated

The

motivation

is

that

if

a

named

entity

occurs

in

many

different

parts

of

a

document

then

it

is

more

likely

to

be

an

important

entity

Therefore

we

use

the

entropy

of

the

probability

distribution

that

measures

how

evenly

an

entity

is

distributed

in

a

document

Consider

a

document

which

is

divided

into

m

sections

Suppose

that

each

named

entity

's

probability

distribution

is

given

by

-LCB-

p

p

i

p

m

-RCB-

where

p

i

occurrence

in

ith

section

total

occurrence

in

the

doc

The

entropy

of

the

named

entity

distribution

is

computed

by

entropy

m

i

p

i

log

p

i

In

our

experiments

we

select

m

This

feature

contributes

a

real

valued

feature-component

to

the

feature

vector

Entity

Neighbor

The

context

in

which

a

certain

named

entity

appears

is

quite

useful

In

this

study

we

only

consider

a

simple

feature

which

counts

its

left

and

right

neighboring

entity

types

If

several

named

entities

of

the

same

type

are

listed

side

by

side

then

it

is

likely

that

the

purpose

is

for

enumeration

and

the

listed

named

entities

are

not

important

Each

neighboring

side

has

five

possible

types

four

named

entity

types

plus

a

normal-word

-LRB-

not

a

named

entity

-RRB-

type

For

example

consider

a

person

mentioned

three

times

in

the

document

Among

the

three

mentions

the

left

neighbors

are

two

person

names

and

one

common

word

and

the

right

neighbors

are

one

place

name

and

two

common

words

Then

the

entity

neighbor

feature

components

are

-LSB-

-RSB-

First

Sentence

Occurrence

This

feature

is

inspired

by

the

position

method

-LSB-

-RSB-

in

sentence

extraction

Its

value

is

the

occurrences

of

the

entity

appearing

in

the

beginning

sentence

of

a

paragraph

Document

Has

Entity

in

Title

or

Not

This

feature

indicates

whether

any

entity

exists

in

the

title

of

the

document

and

thus

takes

binary

value

of

or

Total

Entity

Count

This

feature

is

the

total

number

of

entities

in

the

document

which

takes

integer

value

The

feature

reflects

the

relative

importance

of

an

entity

in

the

entity

set

Document

Frequency

in

the

Corpus

This

is

a

corpus

level

feature

If

a

named

entity

has

a

low

frequency

in

the

document

collection

but

relatively

high

frequency

in

the

current

document

then

it

is

likely

to

be

a

focused

entity

When

this

feature

is

used

the

term

frequency

feature

in

section

will

be

computed

using

-LRB-

tf

docsize

-RRB-

log

-LRB-

N/df

-RRB-

where

df

is

the

number

of

documents

that

a

named

entity

occurs

in

In

this

section

we

study

the

following

issues

corpus

annotation

human-level

agreement

on

focused

named

entities

performance

of

machine

learning

methods

compared

with

a

baseline

influence

of

different

features

and

the

impact

of

coreference

module

to

the

overall

performance

Corpus

Annotation

We

select

fifteen

days

of

Beijing

Youth

Daily

news

in

November

as

our

testing

corpus

which

contains

articles

The

text

downloaded

from

is

in

Chinese

The

articles

belong

to

a

variety

of

categories

including

politics

economy

laws

education

science

entertainments

and

sports

Since

different

people

may

have

different

opinions

on

the

focused

named

entities

a

common

set

of

rules

should

be

agreed

upon

before

the

whole

corpus

is

to

be

annotated

We

use

the

following

method

to

come

up

with

a

general

guideline

for

annotating

focused

named

entities

First

the

named

entities

in

each

document

were

annotated

by

human

Then

we

selected

twenty

documents

from

the

corpus

and

invited

twelve

people

to

mark

focused

named

entities

Nine

of

the

twelve

people

are

experts

in

natural

language

processing

so

their

opinions

are

very

valuable

to

define

focused

named

entities

Based

on

the

survey

result

entities

marked

by

more

than

half

of

the

survey

participants

were

defined

as

focused

named

entities

We

obtained

fifty

focused

named

entities

for

the

twenty

articles

By

studying

the

focused

named

entities

in

those

articles

we

were

able

to

design

specifications

for

focused

named

entity

annotation

The

whole

corpus

was

then

marked

according

to

the

specifications

Human

Agreement

Statistics

In

our

survey

fifty

entities

were

identified

as

focused

entities

from

the

total

number

of

entities

in

the

documents

Table

shows

among

the

focused

entities

entities

are

agreed

as

focus

by

all

persons

and

entities

are

agreed

by

persons

etc.

Table

Human

agreement

statistics

num

of

focused

named

entities

num

of

person

agreeing

Let

N

k

denotes

the

number

of

person

with

agreement

on

focused

named

entity

k

then

the

human

agreement

level

Agree

k

on

the

k-th

focused

named

entity

is

Agree

k

N

k

The

average

agreement

on

the

focused

named

entities

is

Average

Agree

k

Agree

k

with

variance

We

also

computed

the

precision

and

the

recall

for

the

survey

participants

with

respect

to

the

fifty

focused

named

entities

Table

shows

that

the

best

human

annotator

achieves

an

F

measure

of

Some

of

the

participants

marked

either

too

many

or

too

few

named

entities

and

thus

had

much

lower

performance

numbers

This

problem

was

fixed

when

the

whole

corpus

was

annotated

using

specifications

induced

from

this

small-scale

experiment

Table

Human

annotation

performance

user

id

precision

recall

F

Corpus

Named

Entity

Statistics

We

consider

two

data

sets

in

our

experiments

one

is

the

whole

corpus

of

articles

and

the

other

is

a

subset

of

articles

with

named

entities

in

their

titles

Table

shows

there

are

totally

focused

entities

among

entities

in

the

whole

corpus

which

means

that

percent

of

the

entities

are

marked

as

focused

On

average

there

are

focused

named

entities

for

each

article

which

is

consistent

with

the

small-scale

survey

result

Table

Corpus

statistics

on

named

entities

set

docnum

entities

focuses

focus

percent

focus/doc

Baseline

Results

Since

named

entities

in

title

or

with

high

frequency

are

more

likely

to

be

the

focal

entities

we

consider

three

baseline

methods

The

first

method

marks

entities

in

titles

to

be

the

foci

the

second

method

marks

most

frequent

entities

in

each

article

to

be

the

focal

entities

the

third

method

is

a

combination

of

the

above

two

which

selects

those

entities

either

in

title

or

occurring

most

frequently

We

use

partial

string

matching

for

coreference

resolution

in

the

three

baseline

experiments

Named

entities

occurring

in

the

title

are

more

likely

to

be

the

focus

of

the

document

but

they

only

represent

a

small

portion

of

all

focal

entities

Baseline

experiment

shows

the

precision

of

this

method

is

quite

high

but

the

recall

is

very

low

Baseline

experiment

implies

that

most

of

the

top

named

entities

are

focused

entities

but

again

the

recall

is

very

low

However

if

more

named

entities

are

selected

the

precision

is

decreased

significantly

so

that

the

F

measure

does

not

improve

The

performance

is

the

worst

with

an

F

measure

of

only

Note

that

several

named

entities

may

have

the

same

occurrence

frequency

in

one

document

which

introduces

uncertainty

into

the

method

By

combining

named

entities

from

the

title

and

with

high

frequency

we

obtain

better

results

than

either

of

the

two

basic

baseline

methods

The

best

performance

is

achieved

by

combining

the

in-title

and

top

named

entities

which

achieves

F

measures

of

for

data

set

and

for

data

set

Machine

Learning

Results

Since

in

our

implementation

decision

tree

and

naive

Bayes

methods

only

take

integer

features

we

encode

the

floating

features

to

integer

values

using

a

simple

equal

interval

binning

method

If

a

feature

x

is

observed

to

have

values

bounded

by

x

min

and

x

max

then

the

bin

width

is

computed

by

x

max

x

min

k

and

the

bin

boundaries

are

at

x

min

i

where

i

k

The

method

is

applied

to

each

continuous

feature

independently

and

k

is

set

to

Although

more

sophisticated

discretization

methods

exist

the

equal

interval

binning

method

performs

quite

well

in

practice

Machine

learning

results

are

obtained

from

five-fold

cross-validation

Coreference

resolution

is

done

with

partial

string-matching

The

test

results

are

reported

in

Table

This

experiment

shows

that

good

performance

can

be

achieved

by

using

machine

learning

techniques

The

RRM

performance

on

both

data

sets

are

significantly

better

than

the

base

line

results

and

comparable

to

that

of

the

best

human

annotator

we

observed

from

our

small-scale

experiment

in

Section

Table

Baseline

results

Corpus

Method

Focuses

focus/doc

Precision

Recall

F

title

docs

title

title

title

docs

title

title

title

Table

Machine

learning

results

Dataset

RRM

Decision

Tree

Naive

Bayes

P

R

F

P

R

F

P

R

F

docs

docs

Influence

of

Features

The

goal

of

this

section

is

to

study

the

impact

of

different

features

with

different

algorithms

Results

are

reported

in

Table

Feature

id

corresponds

to

the

feature

subsection

number

in

section

Experiment

A

uses

frequency-based

features

only

It

is

quite

similar

to

the

bag-of-word

document

model

for

text

categorization

with

the

entity-frequency

and

in-title

information

By

adding

more

sophisticated

document-level

features

the

performance

can

be

significantly

improved

For

the

RRM

method

F

finally

reaches

It

is

interesting

to

observe

that

the

corpus-level

feature

-LRB-

experiment

F

versus

G

-RRB-

has

different

impacts

on

the

three

algorithms

It

is

a

good

feature

for

naive

Bayes

but

not

for

the

RRM

and

decision

tree

Whether

corpus-level

features

can

appreciably

enhance

the

classification

performance

requires

more

careful

investigation

The

experiments

also

indicate

that

the

three

learning

algorithms

do

not

perform

equally

well

RRM

appears

to

have

the

best

overall

performance

The

naive

Bayes

method

requires

all

features

to

be

independent

which

is

a

quite

unreal-istic

assumption

in

practice

The

main

problem

for

decision

tree

is

that

it

easily

fragments

the

data

so

that

the

probability

estimate

at

the

leaf-nodes

become

unreliable

This

is

also

the

reason

why

voted

decision

trees

-LRB-

using

procedures

like

boosting

or

bagging

-RRB-

perform

better

The

decision

tree

can

find

rules

readable

by

a

human

For

example

one

such

rule

reads

as

if

a

named

entities

appears

at

least

twice

its

left

and

right

neighbors

are

normal

words

its

discrete

distribution

entropy

is

greater

than

and

the

entity

appears

in

the

title

then

the

probability

of

it

being

a

focused

entity

is

Coreference

Resolution

In

order

to

understand

the

impact

of

coreference

resolution

on

the

performance

of

focused

named

entity

recognition

we

did

the

same

set

of

experiments

as

in

section

but

with

exact

string

matching

only

for

coreference

resolution

in

the

feature

extraction

process

Table

reports

the

five-fold

cross

validation

results

On

average

the

performance

is

decreased

by

about

to

percent

This

means

coreference

resolution

plays

an

important

role

in

the

task

The

reason

is

that

it

maps

variations

of

a

named

entity

into

a

single

group

so

that

features

such

as

occurrence

frequency

and

entity

distribution

can

be

estimated

more

reliably

We

believe

that

with

more

sophisticated

analysis

such

as

pronominal

coreference

resolution

the

classification

performance

can

be

further

improved

In

this

paper

we

studied

the

problem

of

focused

named

entity

recognition

We

gave

examples

to

illustrate

that

focused

named

entities

are

useful

for

many

natural

language

processing

applications

The

task

can

be

converted

into

a

binary

classification

problem

We

focused

on

designing

linguistic

features

and

compared

the

performance

of

three

machine

learning

algorithms

Our

results

show

that

the

machine

learning

approach

can

achieve

near

human-level

accuracy

Because

our

system

is

trainable

and

features

we

use

are

language

independent

it

is

easy

for

us

to

build

a

similar

classification

model

for

other

languages

Our

method

can

also

be

generalized

to

related

tasks

such

as

finding

important

words

and

noun-phrases

in

a

document

In

the

future

we

will

integrate

focused

named

entity

recognition

into

real

applications

such

as

information

retrieval

automatic

summarization

and

topic

detection

and

tracking

so

that

we

can

further

study

and

evaluate

its

influences

to

these

systems

We

thank

Honglei

Guo

for

providing

the

original

corpus

with

named

entity

annotation

Jianmin

Jiang

helped

us

to

set

up

named

entity

annotation

server

which

made

it

much

easier

to

view

and

annotate

the

corpus

We

thank

Zhaoming

Qiu

Shixia

Liu

Zhili

Guo

for

their

valuable

comments

on

the

experiments

We

are

also

grateful

to

our

colleagues

who

spent

their

time

to

participate

in

the

focused

named

entity

survey

-LSB-

-RSB-

R.

Barzilay

and

M.

Elhadad

Using

lexical

chains

for

text

summarization

In

Proceedings

of

the

ACL

Table

Performance

of

different

features

with

different

algorithms

ID

Features

RRM

Decision

Tree

Naive

Bayes

P

R

F

P

R

F

P

R

F

A

B

C

D

E

F

G

Table

Machine

learning

test

result

with

exact

string-matching

for

coreference

resolution

data

set

RRM

Decision

Tree

Naive

Bayes

P

R

F

P

R

F

P

R

F

docs

docs

Intelligent

Scalable

Text

Summarization

Workshop

-LRB-

ISTS

-RRB-

pages

-LSB-

-RSB-

F.

J.

Damerau

T.

Zhang

S.

M.

Weiss

and

N.

Indurkhya

Text

categorization

for

a

comprehensive

time-dependent

benchmark

Information

Processing

Management

-LRB-

-RRB-

-LSB-

-RSB-

H.

P.

Edmundson

New

methods

in

automatic

abstracting

Journal

of

The

Association

for

Computing

Machinery

-LRB-

-RRB-

-LSB-

-RSB-

J.

Y.

Ge

X.

J.

Huang

and

L.

Wu

Approaches

to

event-focused

summarization

based

on

named

entities

and

query

words

In

DUC

Workshop

on

Text

Summarization

-LSB-

-RSB-

E.

Hovy

and

C.-Y

Lin

Automated

text

summarization

in

summarist

In

I.

Mani

and

M.

Maybury

editors

Advances

in

Automated

Text

Summarization

pages

MIT

Press

-LSB-

-RSB-

D.

E.

Johnson

F.

J.

Oles

T.

Zhang

and

T.

Goetz

A

decision-tree-based

symbolic

rule

induction

system

for

text

categorization

IBM

Systems

Journal

-LSB-

-RSB-

M.-Y

Kan

and

K.

R.

McKeown

Information

extraction

and

summarization

domain

independence

through

focus

types

Columbia

University

Computer

Science

Technical

Report

-LSB-

-RSB-

J.

M.

Kupiec

J.

Pedersen

and

F.

Chen

A

trainable

document

summarizer

In

SIGIR

pages

-LSB-

-RSB-

D.

Lawrie

W.

B.

Croft

and

A.

Rosenberg

Finding

topic

words

for

hierarchical

summarization

In

SIGIR

pages

-LSB-

-RSB-

F.

Li

and

Y.

Yang

A

loss

function

analysis

for

classification

methods

in

text

categorization

In

ICML

pages

-LSB-

-RSB-

C.-Y

Lin

Training

a

selection

function

for

extraction

In

CIKM

pages

-LSB-

-RSB-

C.-Y

Lin

and

E.

Hovy

Identifying

topics

by

position

In

Proceedings

of

the

Applied

Natural

Language

Processing

Conference

-LRB-

-RRB-

pages

-LSB-

-RSB-

D.

Marcu

From

discourse

structures

to

text

summaries

In

Proceedings

of

the

ACL

Workshop

on

Intelligent

Scalable

Text

Summarization

pages

ACL

-LSB-

-RSB-

A.

McCallum

and

K.

Nigam

A

comparison

of

event

models

for

naive

bayes

text

classification

In

AAAI/ICML

Workshop

on

Learning

for

Text

Categorization

pages

-LSB-

-RSB-

J.

L.

Neto

A.

Santos

C.

Kaestner

A.

Freitas

and

J.

Nievola

A

trainable

algorithm

for

summarizing

news

stories

In

Proceedings

of

PKDD

Workshop

on

Machine

Learning

and

Textual

Information

Access

September

-LSB-

-RSB-

C.

Nobata

S.

Sekine

H.

Isahara

and

R.

Grishman

Summarization

system

integrated

with

named

entity

tagging

and

ie

pattern

discovery

In

Proceedings

of

Third

International

Conference

on

Language

Resources

and

Evaluation

-LRB-

LREC

-RRB-

-LSB-

-RSB-

C.

D.

Paice

and

P.

A.

Jones

The

identification

of

important

concepts

in

highly

structured

technical

papers

In

SIGIR

pages

ACM

-LSB-

-RSB-

J.

R.

Quinlan

Programs

for

Machine

Learning

Morgan

Kaufmann

-LSB-

-RSB-

E.

F.

T.

K.

Sang

and

F.

D.

Meulder

Introduction

to

the

shared

task

Language-independent

named

entity

recognition

In

Proceedings

of

pages

-LSB-

-RSB-

W.-M

Soon

H.-T

Ng

and

C.-Y

Lim

A

machine

learning

approach

to

coreference

resolution

of

noun

phrases

Computational

Linguistics

-LRB-

-RRB-

-LSB-

-RSB-

S.

Teufel

and

M.

Moens

Sentence

extraction

as

a

classification

task

In

ACL/EACL

Workshop

on

Intelligent

and

Scalable

Text

Summarization

-LSB-

-RSB-

T.

Zhang

On

the

dual

formulation

of

regularized

linear

systems

Machine

Learning

-LSB-

-RSB-

T.

Zhang

F.

Damerau

and

D.

E.

Johnson

Text

chunking

based

on

a

generalization

of

Winnow

Journal

of

Machine

Learning

Research

-LSB-

-RSB-

T.

Zhang

and

F.

J.

Oles

Text

categorization

based

on

regularized

linear

classification

methods

Information

Retrieval

O O B I O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O B I O O O O B I I O O O O O B O O O B O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O B O O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O B I I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O B I O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O B I I O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O B O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O

electronic documents
naive bayes
statistical model
entity recognition
natural language processing
classification methods
features
text summarization
machine learning approach
machine learning
decision tree
summarization
information retrieval
sentence extraction
robust risk minimization
named entities
ranking
classification model
main topics

natural language processing applications