Automated

Rich

Presentation

of

a

Semantic

Topic

To

have

a

rich

presentation

of

a

topic

it

is

not

only

expected

that

many

relevant

multimodal

information

including

images

text

audio

and

video

could

be

extracted

it

is

also

important

to

organize

and

summarize

the

related

information

and

provide

users

a

concise

and

informative

storyboard

about

the

target

topic

It

facilitates

users

to

quickly

grasp

and

better

understand

the

content

of

a

topic

In

this

paper

we

present

a

novel

approach

to

automatically

generating

a

rich

presentation

of

a

given

semantic

topic

In

our

proposed

approach

the

related

multimodal

information

of

a

given

topic

is

first

extracted

from

available

multimedia

databases

or

websites

Since

each

topic

usually

contains

multiple

events

a

text-based

event

clustering

algorithm

is

then

performed

with

a

generative

model

Other

media

information

such

as

the

representative

images

possibly

available

video

clips

and

flashes

-LRB-

interactive

animates

-RRB-

are

associated

with

each

related

event

A

storyboard

of

the

target

topic

is

thus

generated

by

integrating

each

event

and

its

corresponding

multimodal

information

Finally

to

make

the

storyboard

more

expressive

and

attractive

an

incidental

music

is

chosen

as

background

and

is

aligned

with

the

storyboard

A

user

study

indicates

that

the

presented

system

works

quite

well

on

our

testing

examples

H.

-LSB-

Information

Interfaces

and

Presentation

-RSB-

Group

and

Organization

Interfaces

Organizational

design

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Content

Analysis

and

Indexing

abstracting

methods

Algorithms

Design

Management

Experimentation

Theory

In

the

multimedia

field

a

major

objective

of

content

analysis

is

to

discover

the

high-level

semantics

and

structures

from

the

low-level

features

and

thus

to

facilitate

indexing

browsing

searching

and

managing

the

multimedia

database

In

recent

years

a

lot

of

technologies

have

been

developed

for

various

media

types

including

images

video

audio

and

etc.

For

example

various

approaches

and

systems

have

been

proposed

in

image

content

analysis

such

as

semantic

classification

-LSB-

-RSB-

content-based

image

retrieval

-LSB-

-RSB-

and

photo

album

management

-LSB-

-RSB-

There

are

also

a

lot

of

research

focuses

on

video

analysis

such

as

video

segmentation

-LSB-

-RSB-

highlight

detection

-LSB-

-RSB-

video

summarization

-LSB-

-RSB-

-LSB-

-RSB-

and

video

structure

analysis

-LSB-

-RSB-

applied

in

various

data

including

news

video

movie

and

sports

video

Since

audio

information

is

very

helpful

for

video

analysis

many

research

works

on

audio

are

also

developed

to

enhance

multimedia

analysis

such

as

audio

classification

-LSB-

-RSB-

and

audio

effect

detection

in

different

audio

streams

-LSB-

-RSB-

Most

recently

there

are

more

and

more

approaches

and

systems

integrating

multimodal

information

in

order

to

improve

analysis

performance

-LSB-

-RSB-

-LSB-

-RSB-

The

main

efforts

of

the

above

mentioned

research

have

focused

on

understanding

the

semantics

-LRB-

including

a

topic

an

event

or

the

similarity

-RRB-

from

the

multimodal

information

That

is

after

the

multimedia

data

is

given

we

want

to

detect

the

semantics

implied

in

these

data

In

this

paper

we

propose

a

new

task

Rich

Presentation

which

is

an

inverse

problem

of

the

traditional

multimedia

content

analysis

That

is

if

we

have

a

semantic

topic

how

can

we

integrate

its

relevant

multimodal

information

including

image

text

audio

and

video

to

richly

present

the

target

topic

and

to

provide

users

a

concise

and

informative

storyboard

In

this

paper

the

so-called

semantic

topic

is

a

generic

concept

It

could

be

any

keyword

representing

an

event

or

events

a

person

's

name

or

anything

else

For

example

World

Cup

and

US

election

could

be

topics

as

well

as

Halloween

and

Harry

Potter

In

this

paper

our

task

is

to

find

sufficient

information

on

these

topics

extract

the

key

points

fuse

the

information

from

different

modalities

and

then

generate

an

expressive

storyboard

Rich

presentation

can

be

very

helpful

to

facilitate

quickly

grasping

and

better

understanding

the

corresponding

topic

People

usually

search

information

from

-LRB-

multimedia

-RRB-

database

or

the

Internet

However

what

they

get

is

usually

a

bulk

of

unorganized

information

with

many

duplicates

and

noise

It

is

tedious

and

costs

a

long

time

to

get

what

they

want

by

browsing

the

search

results

If

there

is

a

tool

to

help

summarize

and

integrate

the

multimodal

information

and

then

produce

a

concise

and

informative

storyboard

it

will

enable

users

to

quickly

figure

out

the

overview

contents

of

a

topic

that

they

want

to

understand

Rich

presentation

provides

such

a

tool

and

thus

it

could

have

many

potential

applications

such

as

education

and

learning

multimedia

authoring

multimedia

retrieval

documentary

movie

production

and

information

personalization

In

this

paper

we

will

present

the

approach

to

rich

presentation

In

order

to

produce

a

concise

and

informative

storyboard

to

richly

present

a

target

topic

we

need

to

answer

the

following

questions

-RRB-

How

to

extract

the

relevant

information

regarding

the

target

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

MM

November

Singapore

Copyright

ACM

topic

-RRB-

How

to

extract

the

key

points

from

the

relevant

information

and

build

a

concise

and

informative

storyboard

-RRB-

How

to

fuse

all

the

information

from

different

modality

and

-RRB-

how

to

design

the

corresponding

rendering

interface

Storyboard

Relevant

Media

Music

Multiple

Events

Clustering

Event

summary

-LRB-

time

-RRB-

Geographic

information

Relevant

multimodal

information

Retrieval

Text

A

Target

Topic

Rhythm

Analysis

Onset/Beat

Sequence

Strength

confidence

Media

Association

Representative

images

Relevant

video

clips

Storyboard

Generation

Event

presentation

multimodal

information

fusion

layout

design

Music

and

storyboard

synchronization

Rich

Presentation

User

Interaction

Fig.

The

system

framework

of

rich

presentation

of

a

target

semantic

topic

It

is

mainly

composed

of

three

steps

relevant

multimodal

information

extraction

media

analysis

and

rich

presentation

generation

In

this

paper

we

propose

a

number

of

novel

approaches

to

deal

with

the

above

issues

and

also

present

an

example

system

Fig.

illustrates

the

proposed

system

framework

of

rich

presentation

It

is

mainly

composed

of

three

steps

relevant

multimodal

information

extraction

media

analysis

including

multiple

events

clustering

representative

media

detection

and

music

rhythm

analysis

and

the

final

storyboard

generation

and

music

synchronization

In

the

proposed

system

given

the

semantic

topic

the

relevant

information

including

text

image

video

and

music

is

first

extracted

from

the

available

multimedia

database

or

the

web

database

User

interaction

is

also

allowed

to

provide

extra

relevant

material

or

give

relevant

feedback

Then

the

information

is

summarized

with

an

event

clustering

algorithm

to

give

a

concise

representation

of

the

topic

and

figure

out

the

overview

of

the

contents

Other

multimedia

materials

such

as

representative

images

-LRB-

or

image

sequences

-RRB-

and

geographic

information

are

subsequently

associated

with

each

event

In

the

next

step

all

the

above

information

is

integrated

to

generate

a

storyboard

in

which

each

event

is

presented

as

one

or

multiple

slides

An

incidental

music

which

is

also

possibly

relevant

to

the

topic

is

finally

synchronized

with

the

storyboard

to

improve

its

expressiveness

and

attractiveness

Thus

with

these

steps

a

concise

and

informative

rich

presentation

regarding

the

target

topic

is

generated

The

rest

of

the

paper

is

organized

as

follows

Section

discusses

the

relevant

information

extraction

corresponding

to

the

target

topic

Section

presents

our

approach

to

the

topic

representation

including

multiple

events

clustering

event

description

and

representative

media

selection

Section

describes

the

approach

to

rich

presentation

generation

including

storyboard

generation

incidental

music

analysis

and

synchronization

Experiments

and

evaluations

are

presented

in

the

Section

Conclusions

are

given

in

the

Section

To

obtain

the

multimodal

information

which

is

relevant

to

the

input

topic

-LRB-

keyword

-RRB-

generally

we

could

search

them

from

various

databases

which

have

been

indexed

with

the

state-of-the-art

multimedia

analysis

techniques

However

in

current

stage

there

is

lack

of

such

publicly

available

multimedia

databases

The

public

search

engine

like

MSN

or

Google

indexes

all

the

Internet

web-pages

and

can

return

a

lot

of

relevant

information

but

the

search

results

usually

contain

much

noise

We

could

also

build

a

private

database

for

this

system

to

provide

more

relevant

and

clean

results

but

it

will

be

too

much

expensive

to

collect

and

annotate

sufficient

multimedia

data

for

various

topics

In

order

to

obtain

relatively

accurate

and

sufficient

data

for

an

arbitrary

topic

in

our

system

we

chose

to

collect

the

relevant

multimodal

information

of

the

given

topic

from

the

news

websites

such

as

MSNBC

BBC

and

CNN

instead

of

building

an

available

database

from

the

scratch

These

news

websites

are

usually

well

organized

and

managed

and

contain

various

kinds

of

high

quality

information

including

text

image

and

news

video

clips

Although

the

news

websites

are

used

as

the

information

sources

in

our

system

other

various

multimedia

databases

can

be

also

easily

incorporated

into

the

system

if

they

are

available

Instead

of

directly

submitting

the

topic

as

a

query

and

getting

the

returned

results

by

using

the

search

function

provided

by

the

websites

in

our

system

we

crawled

the

news

documents

from

these

websites

in

advance

and

then

build

a

full-text

index

It

enables

us

to

quickly

obtain

the

relevant

documents

and

also

enable

us

to

use

some

traditional

information

retrieval

technologies

such

as

query

expansion

-LSB-

-RSB-

to

remove

the

query

ambiguousness

and

get

more

relevant

documents

In

our

approach

user

interaction

is

also

allowed

to

provide

more

materials

relevant

to

the

topic

or

give

relevant

feedback

on

the

returned

results

For

example

from

the

above

websites

we

can

seldom

find

a

music

clip

relevant

to

the

target

topic

In

this

case

users

could

provide

the

system

a

preferred

music

which

will

be

further

used

as

incidental

music

to

accompany

with

the

storyboard

presentation

Users

could

also

give

some

feedbacks

on

the

obtained

documents

For

example

if

he

gives

a

thumb-up

to

a

document

the

relevant

information

of

the

document

needs

to

be

presented

in

the

final

storyboard

On

the

other

side

users

could

also

thumb-down

a

document

to

remove

the

related

information

A

semantic

topic

is

usually

a

quite

broad

concept

and

it

usually

contains

multiple

events

For

example

in

the

topic

Harry

Potter

the

publication

of

each

book

and

the

release

of

each

movie

could

be

considered

as

an

event

while

in

the

topic

World

Cup

each

match

could

also

be

taken

as

an

event

For

each

event

there

are

usually

many

documents

reporting

it

Therefore

in

order

to

generate

an

informative

and

expressive

storyboard

to

present

the

topic

it

would

be

better

to

decompose

the

obtained

information

and

cluster

the

documents

into

different

events

However

event

definition

is

usually

subjective

different

individuals

may

have

different

opinions

It

is

also

confusing

in

which

scale

an

event

should

be

defined

Also

take

World

Cup

as

an

example

in

a

larger

scale

World

Cup

and

World

Cup

could

also

be

considered

as

a

big

event

Therefore

due

to

the

above

vagueness

in

this

paper

we

do

not

strictly

define

each

event

of

the

target

topic

Following

our

previous

works

on

news

event

detection

-LSB-

-RSB-

an

event

is

assumed

as

some

similar

information

describing

similar

persons

similar

keywords

similar

places

and

similar

time

duration

Therefore

in

our

system

an

event

is

represented

by

four

primary

elements

who

-LRB-

persons

-RRB-

when

-LRB-

time

-RRB-

where

-LRB-

locations

-RRB-

and

what

-LRB-

keywords

-RRB-

and

event

clustering

is

to

group

the

documents

reporting

similar

primary

elements

As

for

the

scale

of

event

in

the

paper

it

could

be

adaptively

determined

by

the

time

range

of

the

obtained

documents

or

the

required

event

number

In

this

section

we

present

a

novel

clustering

approach

based

on

a

generative

model

proposed

in

-LSB-

-RSB-

instead

of

using

traditional

clustering

methods

such

as

K-means

After

event

clusters

are

obtained

the

corresponding

event

summary

is

then

extracted

and

other

representative

media

is

associated

with

each

event

Multiple

Event

Clustering

To

group

the

documents

into

different

events

essentially

we

need

to

calculate

p

-LRB-

e

j

x

i

-RRB-

which

represents

the

probability

that

a

document

x

i

belongs

to

an

event

e

j

Here

as

mentioned

above

an

event

e

j

-LRB-

and

thus

the

document

x

i

describing

the

event

-RRB-

is

represented

by

four

primary

elements

who

-LRB-

persons

-RRB-

when

-LRB-

time

-RRB-

where

-LRB-

locations

-RRB-

and

what

-LRB-

keywords

-RRB-

That

is

-RCB-

-LCB-

time

keywords

locations

persons

Docment

Event

Assuming

that

a

document

is

always

caused

by

an

event

-LSB-

-RSB-

and

the

four

primary

elements

are

independent

to

calculate

the

probability

p

-LRB-

e

j

x

i

-RRB-

in

our

approach

we

first

determine

the

likelihood

that

the

document

x

i

is

generated

from

event

e

j

p

-LRB-

x

i

e

j

-RRB-

which

could

be

further

represented

by

the

following

generative

model

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

j

i

j

i

j

i

j

i

j

i

e

time

p

e

key

p

e

loc

p

e

name

p

e

x

p

-LRB-

-RRB-

where

name

i

loc

i

key

i

and

time

i

are

the

feature

vectors

representing

persons

locations

keywords

and

time

in

the

document

x

i

respectively

In

our

approach

the

above

entities

are

extracted

by

the

BBN

NLP

tools

-LSB-

-RSB-

The

tool

can

extract

seven

types

of

entities

including

persons

organizations

locations

date

time

money

and

percent

In

our

approach

the

obtained

organization

entity

is

also

considered

as

a

person

entity

and

all

the

words

except

of

persons

locations

and

other

stop-words

are

taken

as

keywords

In

more

detail

name

i

-LRB-

similarly

loc

i

and

key

i

-RRB-

is

a

vector

c

c

c

iNp

where

c

in

is

the

occurrence

frequency

of

the

person

n

appears

in

the

document

x

i

and

person

n

is

the

nth

person

in

the

person

vocabulary

which

is

composed

of

all

the

persons

appeared

in

all

the

obtained

documents

-LRB-

similarly

we

can

define

keyword

vocabulary

and

location

vocabulary

-RRB-

Assuming

N

p

is

the

size

of

person

vocabulary

p

-LRB-

name

i

e

j

-RRB-

could

be

further

expressed

by

p

in

N

n

c

j

n

j

i

e

person

p

e

name

p

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

Since

the

person

location

and

keyword

are

discrete

variables

represented

by

words

and

the

probability

of

the

location

and

keyword

can

be

also

defined

similarly

as

that

of

the

person

in

-LRB-

-RRB-

in

the

flowing

sections

we

will

not

discriminate

them

and

uniformly

represent

the

probability

p

-LRB-

person

n

e

j

-RRB-

-LRB-

correspond-ingly

the

p

-LRB-

location

n

e

j

-RRB-

and

p

-LRB-

keyword

n

e

j

-RRB-

-RRB-

as

p

-LRB-

w

n

e

j

-RRB-

which

denotes

the

probability

that

the

word

w

n

appears

in

the

event

e

j

On

the

other

hand

the

time

of

an

event

usually

lasts

a

continuous

duration

It

is

also

observed

especially

in

the

news

domain

that

the

documents

about

an

event

usually

increases

at

the

beginning

stage

of

the

event

and

then

decreases

at

the

end

Therefore

in

our

approach

a

Gaussian

model

N

-LRB-

u

j

j

-RRB-

is

utilized

to

roughly

represent

the

probability

p

-LRB-

time

i

e

j

-RRB-

where

u

j

and

j

is

the

mean

and

standard

deviation

respectively

To

this

end

in

order

to

estimate

the

probability

p

-LRB-

e

j

x

i

-RRB-

we

need

to

estimate

the

parameters

-LCB-

p

-LRB-

w

n

e

j

-RRB-

u

j

j

-RCB-

assuming

K

is

the

number

of

events

-LRB-

the

selection

of

K

is

discussed

in

section

-RRB-

In

our

approach

the

Maximum

Likelihood

is

used

to

estimate

the

model

parameters

as

M

i

K

j

j

i

j

M

i

i

e

x

p

e

p

x

p

X

p

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

log

-LRB-

max

arg

-RRB-

-RRB-

-LRB-

log

-LRB-

max

arg

-RRB-

-RRB-

-LRB-

log

-LRB-

max

arg

-LRB-

-RRB-

where

X

represents

the

corpus

of

the

obtained

documents

M

and

K

are

number

of

documents

and

events

respectively

Since

it

is

difficult

to

derive

a

close

formula

to

estimate

the

parameters

in

our

approach

an

Expectation

Maximization

-LRB-

EM

-RRB-

algorithm

is

applied

to

maximize

the

likelihood

by

running

E-step

and

M-step

iteratively

A

brief

summary

of

these

two

steps

is

listed

as

follows

and

more

details

can

be

found

in

-LSB-

-RSB-

In

E-step

the

posterior

probability

p

-LRB-

e

j

x

i

-RRB-

is

estimated

as

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

i

t

j

t

j

i

t

i

j

x

p

e

p

e

x

p

x

e

p

-LRB-

-RRB-

where

the

upper

script

-LRB-

t

-RRB-

indicate

the

tth

iteration

In

M-step

the

model

parameters

are

updated

as

M

i

N

s

t

i

j

M

i

t

i

j

t

j

n

s

i

tf

x

e

p

N

n

i

tf

x

e

p

e

w

p

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

M

i

t

i

j

M

i

i

t

i

j

t

j

x

e

p

time

x

e

p

u

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

M

i

t

i

j

M

i

tj

i

t

i

j

t

j

x

e

p

u

time

x

e

p

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

where

tf

-LRB-

i

n

-RRB-

is

the

term

frequency

of

the

word

w

n

in

the

document

x

i

and

N

is

the

corresponding

vocabulary

size

It

is

noted

that

in

-LRB-

-RRB-

the

Laplace

smoothing

-LSB-

-RSB-

is

applied

to

prevent

zero

probability

for

the

infrequently

occurring

word

At

last

the

prior

of

each

event

is

updated

as

M

x

e

p

e

p

M

i

t

i

j

t

j

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

The

algorithm

can

increase

the

log-likelihood

consistently

with

the

iterations

and

then

converge

to

a

local

maximum

Once

the

parameters

are

estimated

we

can

simply

assign

each

document

to

an

event

as

following

-RRB-

-RRB-

-LRB-

-LRB-

max

arg

i

j

j

i

x

e

p

y

-LRB-

-RRB-

where

y

i

is

the

event

label

of

the

document

x

i

The

advantage

of

this

generative

approach

is

that

it

not

only

considers

the

temporal

continuity

of

an

event

it

also

can

deal

with

the

issue

that

some

events

overlap

in

some

time

durations

In

this

case

the

Gaussian

model

of

the

event

time

can

also

be

overlapped

through

this

data-driven

parameter

estimation

From

this

view

the

event

clustering

is

also

like

a

Gaussian

mixture

model

-LRB-

GMM

-RRB-

estimation

in

the

timeline

Determining

the

Number

of

Events

In

the

above

approach

to

event

clustering

the

event

number

K

is

assumed

known

-LRB-

as

shown

in

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

However

the

event

number

is

usually

very

difficult

to

be

determined

a

priori

In

our

approach

an

intuitive

way

is

adopted

to

roughly

estimate

the

event

number

based

on

the

document

distribution

along

with

the

timeline

As

mentioned

above

it

is

assumed

that

each

document

is

caused

by

an

event

and

the

document

number

of

an

event

changes

with

the

development

of

the

event

According

to

this

property

each

peak

-LRB-

or

the

corresponding

contour

-RRB-

of

the

document

distribution

curve

might

indicate

one

event

-LSB-

-RSB-

as

the

Fig.

shows

Thus

we

can

roughly

estimate

the

event

number

by

simply

counting

the

peak

number

However

the

curve

is

quite

noisy

and

there

inevitably

exist

some

noisy

peaks

in

the

curve

In

order

to

avoid

the

noisy

peaks

in

our

approach

only

the

salient

peaks

are

assumed

to

be

relevant

to

the

event

number

To

detect

the

salient

peaks

we

first

smooth

the

document

curve

with

a

half-Hamming

-LRB-

raised-cosine

-RRB-

window

and

then

remove

the

very

small

peaks

with

a

threshold

Fig.

illustrates

a

smoothed

document

distribution

with

the

corresponding

threshold

collected

on

the

topic

US

Election

in

four

months

In

experiments

the

threshold

is

adaptively

set

as

d

d

where

d

and

d

are

the

mean

and

standard

deviation

of

the

curve

respectively

After

the

smoothing

and

tiny

peaks

removal

we

further

detect

the

valleys

between

every

two

contingent

peaks

Thus

the

range

of

an

event

-LRB-

which

is

correlated

to

the

corresponding

peak

-RRB-

can

be

considered

as

the

envelope

in

the

two

valleys

As

shown

in

the

duration

denoted

by

L

i

R

i

is

a

rough

range

of

the

event

correlated

to

the

peak

P

i

Assuming

an

important

event

usually

has

more

documents

and

has

effects

in

a

longer

duration

the

saliency

of

each

peak

is

defined

as

-RRB-

-RRB-

-LRB-

-LRB-

avr

i

i

avr

i

i

D

R

L

P

P

S

-LRB-

-RRB-

where

P

i

is

the

ith

peak

L

i

and

R

i

is

the

duration

from

the

ith

peak

to

the

previous

and

next

valley

P

avr

is

the

average

peak

value

and

D

avr

is

average

duration

between

two

valleys

in

the

curve

S

i

is

the

saliency

value

of

the

peak

P

i

It

could

also

be

considered

as

the

normalized

area

under

peak

P

i

and

thus

it

roughly

represents

the

document

number

of

the

corresponding

event

In

our

approach

the

top

K

salient

peaks

are

selected

to

determine

the

event

number

-RCB-

-LCB-

max

arg

i

N

i

k

i

i

k

S

S

K

-LRB-

-RRB-

where

i

S

is

the

sorted

saliency

value

from

large

to

small

N

is

total

number

of

detected

peaks

and

is

a

threshold

In

our

experiments

is

set

as

which

roughly

means

that

at

least

documents

will

be

kept

in

the

further

initialization

of

event

clustering

This

selection

scheme

is

designed

to

guarantee

there

is

no

important

information

is

missed

in

presentation

After

the

event

number

and

initial

clusters

-LRB-

the

most

salient

peaks

with

their

corresponding

range

-RRB-

are

selected

the

event

parameters

could

be

initialized

and

then

updated

iteratively

#Doc

Threshold

P

i

P

i

P

L

i

R

i

Peaks

relevant

to

event

Fig.

Peak

saliency

definition

It

also

illustrates

the

smoothed

document

distribution

-LRB-

document

number

per

day

-RRB-

with

the

corresponding

threshold

for

tiny

peak

removal

Each

peak

P

i

is

assumed

to

be

correlated

with

each

event

It

is

noted

that

some

technology

such

as

Bayesian

Information

Criteria

-LRB-

BIC

-RRB-

or

minimum

description

length

-LRB-

MDL

-RRB-

-LSB-

-RSB-

could

be

used

to

estimate

the

optimal

event

number

by

searching

through

a

reasonable

range

of

the

event

number

to

find

the

one

which

maximizes

the

likelihood

in

-LRB-

-RRB-

However

these

algorithms

take

long

time

and

it

is

usually

not

necessary

to

estimate

the

exact

event

number

in

our

scenario

of

rich

presentation

Actually

in

our

system

the

most

important

point

of

event

clustering

is

that

the

clustered

documents

really

represent

the

same

event

rather

than

the

event

number

as

observed

in

the

experiments

Moreover

in

the

step

of

synchronization

between

the

music

and

storyboard

-LRB-

in

the

section

-RRB-

the

number

of

presented

events

may

be

further

refined

based

on

the

user

's

preference

in

order

to

match

the

presentation

duration

with

the

music

duration

Event

Description

After

obtaining

the

events

and

the

corresponding

documents

we

not

only

need

a

concise

event

summary

but

also

need

to

extract

some

representative

media

to

describe

each

event

Event

Summary

A

simple

way

to

summarize

an

event

is

to

choose

some

representative

words

on

the

persons

locations

and

keywords

of

the

event

For

example

for

the

event

e

j

the

leading

actor

could

be

chosen

as

the

person

with

the

maximum

p

-LRB-

person

n

e

j

-RRB-

while

the

major

location

could

be

selected

based

on

p

-LRB-

location

n

e

j

-RRB-

However

such

brief

description

might

have

a

bad

readability

Therefore

in

order

to

increase

the

readability

of

the

summary

in

our

system

we

also

provide

an

alterative

way

That

is

we

choose

a

candidate

document

to

represent

an

event

For

example

the

document

with

the

highest

p

-LRB-

x

i

e

j

-RRB-

is

a

good

candidate

representative

of

the

event

e

j

However

a

document

might

be

too

long

to

be

shown

on

the

storyboard

Therefore

in

our

system

only

the

title-brow

-LRB-

the

text

between

the

news

title

and

news

body

-RRB-

of

the

document

which

usually

exists

and

is

usually

a

good

overview

-LRB-

summary

-RRB-

of

the

document

based

on

our

observation

-LRB-

especially

true

in

our

case

of

news

document

-RRB-

is

selected

to

describe

the

event

I

III

II

IV

Fig.

The

event

template

of

the

Storyboard

which

illustrates

-LRB-

I

-RRB-

the

representative

media

-LRB-

II

-RRB-

geographic

information

-LRB-

III

-RRB-

event

summary

and

-LRB-

IV

-RRB-

a

film

strip

giving

an

overview

of

the

events

in

the

temporal

order

Extracting

Representative

Media

In

the

obtained

documents

describing

an

event

there

are

usually

many

illustrational

images

with

possible

flashes

and

video

clips

These

media

information

is

also

a

good

representative

of

the

corresponding

event

However

since

the

obtained

documents

are

directly

crawled

from

the

news

websites

they

usually

contain

many

noisy

multimedia

resources

such

as

the

advertisements

Moreover

there

also

possible

exist

some

duplicate

images

in

different

documents

describing

the

same

event

Therefore

to

extract

the

representative

media

from

the

documents

we

need

to

remove

noisy

media

and

possible

duplicate

images

Before

this

we

also

performed

a

pre-filtering

to

remove

all

the

images

smaller

than

pixels

in

height

or

width

Noisy

Media

Detection

In

our

approach

a

simple

but

efficient

rule

is

used

to

remove

the

noisy

media

resources

We

find

almost

all

advertisements

are

provided

by

other

agencies

rather

than

these

news

websites

themselves

That

is

the

hosts

of

advertisement

resources

are

from

different

websites

Thus

in

our

approach

we

extract

the

host

names

from

the

URLs

of

all

multimedia

resources

and

remove

those

resources

with

different

host

name

Duplicate

Detection

A

number

of

image

signature

schemes

can

be

adopted

here

to

accomplish

duplicate

detection

In

our

implementation

each

image

is

converted

into

grayscale

and

down-sampled

to

That

is

a

signature

for

each

image

is

obtained

Then

the

Euclidean

distance

of

the

signature

are

taken

as

the

dissimilarity

measure

Images

have

sufficiently

small

distance

are

considered

as

duplicates

Once

removing

the

noisy

resources

and

duplicate

images

we

simply

select

the

large

images

from

the

top

representative

documents

-LRB-

with

the

top

largest

p

-LRB-

x

i

e

j

-RRB-

-RRB-

and

take

them

as

representative

media

of

the

corresponding

event

The

exact

number

of

the

selected

images

is

dependent

on

the

document

number

-LRB-

i.e.

the

importance

-RRB-

of

the

event

and

the

total

image

number

the

event

has

It

is

noted

that

in

our

current

system

we

only

associates

images

with

each

event

However

other

media

like

video

and

flashes

can

be

chosen

in

a

similar

way

In

the

proposed

system

the

above

obtained

information

including

event

summary

and

representative

media

are

fused

to

generate

a

concise

and

informative

storyboard

in

order

to

richly

present

the

target

topic

In

this

section

we

will

first

describe

the

storyboard

generation

for

the

target

topic

by

presenting

each

event

with

the

multimodal

information

Then

we

present

the

approach

to

synchronizing

the

storyboard

with

an

incidental

music

Storyboard

Generation

In

our

approach

a

storyboard

of

a

target

topic

is

generated

by

presenting

each

event

of

the

topic

slide

by

slide

To

describe

an

event

we

have

obtained

the

corresponding

information

including

the

person

time

location

event

summary

and

other

relevant

images

Therefore

to

informatively

present

each

event

we

need

first

to

design

an

event

template

-LRB-

i.e.

an

interface

-RRB-

to

integrate

all

the

information

Fig.

illustrates

the

event

template

used

in

our

proposed

system

with

an

example

event

in

the

topic

US

Election

First

the

template

presents

the

representative

images

in

the

largest

area

-LRB-

part

I

-RRB-

since

the

pictures

are

more

vivid

than

the

words

As

for

each

representative

picture

the

title

and

date

of

the

document

from

which

it

is

extracted

is

also

illustrated

In

the

Fig.

there

are

pictures

extracted

from

documents

Then

the

corresponding

event

summaries

of

these

three

documents

are

presented

-LRB-

part

III

-RRB-

where

each

paragraph

refers

to

the

summary

of

one

document

If

a

user

is

interested

in

one

document

he

can

click

on

the

corresponding

title

to

read

more

details

Moreover

the

geographic

information

of

the

event

is

shown

with

a

map

in

the

top-left

corner

-LRB-

part

II

-RRB-

to

give

users

a

view

of

the

event

location

The

map

is

obtained

from

MapPoint

Location

service

-LSB-

-RSB-

which

can

return

a

corresponding

map

based

on

user

's

location

query

However

the

mapping

is

usually

difficult

especially

when

the

event

location

is

confusing

so

that

the

representative

location

is

not

accurately

detected

For

example

the

event

shown

in

the

Fig

is

mapped

to

Washington

D.C.

rather

than

New

York

where

the

republic

convention

is

held

since

Washington

is

the

most

frequently

mentioned

places

in

the

documents

Finally

a

film

strip

-LRB-

part

IV

-RRB-

is

also

presented

arranging

each

event

in

the

temporal

order

where

each

event

is

simply

represented

by

a

cluster

of

images

with

the

current

event

highlighted

It

enables

users

to

have

a

quick

overview

of

the

past

and

the

future

in

the

event

sequence

By

connecting

various

events

slide

by

slide

we

could

get

an

informative

storyboard

regarding

the

target

topic

In

order

to

catch

the

development

process

of

a

topic

the

events

are

ordered

by

their

timestamps

in

the

generated

storyboard

Synchronizing

with

Music

To

make

the

storyboard

more

expressive

and

attractive

and

to

provide

a

more

relaxing

way

to

read

information

in

the

proposed

system

we

will

accompany

the

storyboard

with

an

incidental

music

and

align

the

transitions

between

event

slides

with

the

music

beats

following

the

idea

in

music

video

generation

-LSB-

-RSB-

-LSB-

-RSB-

Sometimes

music

could

also

provide

extra

information

about

the

target

topic

For

example

when

the

target

topic

is

a

movie

the

corresponding

theme

song

could

be

chosen

for

the

rich

presentation

In

this

sub-section

we

will

present

our

approach

to

music

analysis

and

synchronization

with

the

storyboard

Music

Rhythm

Analysis

In

the

proposed

system

we

detect

the

onset

sequences

instead

of

the

exact

beat

series

to

represent

music

rhythm

This

is

because

the

beat

information

is

sometimes

not

obvious

especially

in

light

music

which

is

usually

selected

as

incidental

music

The

strongest

onset

in

a

time

window

could

be

assumed

as

a

beat

This

is

reasonable

since

there

are

some

beat

positions

in

a

time

window

-LRB-

for

example

seconds

-RRB-

thus

the

most

possible

position

of

a

beat

is

the

position

of

the

strongest

onset

The

process

of

onset

estimation

is

illustrated

in

Fig.

After

FFT

is

performed

on

each

frame

of

an

octave-scale

filter-bank

is

used

to

divide

the

frequency

domain

into

six

sub-bands

including

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RSB-

where

refers

to

the

sampling

rate

Acoustic

Music

Data

FFT

Sub-Band

Envelope

Extractor

Difference

curve

Onset

Curve

Sub-Band

N

Envelope

Extractor

Difference

curve

Fig.

The

process

of

onset

sequence

estimation

After

the

amplitude

envelope

of

each

sub-band

is

extracted

by

using

a

half-Hamming

window

a

Canny

operator

is

used

for

onset

sequence

detection

by

estimating

its

difference

function

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

n

C

n

A

n

D

i

i

-LRB-

-RRB-

where

D

i

-LRB-

n

-RRB-

is

the

difference

function

in

the

ith

sub-band

A

i

-LRB-

n

-RRB-

is

the

amplitude

envelope

of

the

ith

sub-band

and

C

-LRB-

n

-RRB-

is

the

Canny

operator

with

a

Gaussian

kernel

-RSB-

-LSB-

-RRB-

-LRB-

c

c

i

L

L

n

e

i

n

C

-LRB-

-RRB-

where

L

c

is

the

length

of

the

Canny

operator

and

is

used

to

control

the

operator

's

shape

which

are

set

as

and

in

our

implementation

respectively

Finally

the

sum

of

the

difference

curves

of

these

six

sub-bands

is

used

to

extract

onset

sequence

Each

peak

is

considered

as

an

onset

and

the

peak

value

is

considered

as

the

onset

strength

Based

on

the

obtained

onsets

an

incidental

music

is

further

segmented

into

music

sub-clips

where

a

strong

onset

is

taken

as

the

boundary

of

a

music

sub-clip

These

music

sub-clips

are

then

used

as

the

basic

timeline

for

the

synchronization

in

the

next

step

Thus

to

satisfy

the

requirement

that

the

event

slide

transitions

of

the

storyboard

should

occur

at

the

music

beats

we

just

need

to

align

the

event

slide

boundaries

and

music

sub-clip

boundaries

To

give

a

more

pleasant

perception

the

music

sub-clip

should

not

be

too

short

or

too

long

also

it

had

better

not

always

keep

the

same

length

In

our

implementation

the

length

of

music

sub-clips

is

randomly

selected

in

a

range

of

-LSB-

t

min

t

max

-RSB-

seconds

Thus

the

music

sub-clips

can

be

extracted

in

the

following

way

given

the

previous

boundary

the

next

boundary

is

selected

as

the

strongest

onset

in

the

window

which

is

-LSB-

t

min

t

max

-RSB-

seconds

away

from

the

previous

boundary

In

the

proposed

system

users

can

manually

specify

the

range

of

the

length

of

the

music

sub-clip

The

default

range

in

the

system

is

set

as

-LSB-

-RSB-

seconds

in

order

to

let

users

have

enough

time

to

read

all

the

information

on

each

event

slide

Alignment

Scheme

To

synchronize

the

transitions

between

different

event

slides

and

the

beats

of

the

incidental

music

as

mentioned

above

we

actually

need

to

align

the

slide

boundaries

and

music

sub-clip

boundaries

To

satisfy

this

requirement

a

straightforward

way

is

to

set

the

length

of

each

event

slide

be

equal

to

the

corresponding

length

of

the

sub-music

clip

However

as

Fig.

illustrates

the

number

of

event

slides

is

usually

not

equal

to

the

number

of

music

sub-clip

In

this

case

in

our

proposed

system

we

provide

two

schemes

to

solve

this

problem

-RRB-

Music

Sub-clip

Based

In

this

scheme

only

the

top

N

important

events

of

the

target

topic

are

adaptively

chosen

and

used

in

the

rich

presentation

where

N

is

supposed

as

the

number

of

music

sub-clip

in

the

corresponding

incidental

music

as

the

Fig.

shows

Although

a

formal

definition

of

event

importance

is

usually

hard

and

subjective

in

our

approach

the

importance

score

of

an

event

is

simply

measured

by

the

number

of

documents

reporting

it

assuming

that

the

more

important

the

event

the

more

the

corresponding

documents

The

assumption

is

quite

similar

as

that

in

the

definition

of

-LRB-

-RRB-

-RRB-

Specified

Event

Number

Based

In

this

scheme

users

can

specify

the

number

of

the

event

he

wants

to

learn

For

example

a

user

could

choose

to

show

the

top

important

events

or

all

the

events

Thus

to

accommodate

all

the

events

in

the

music

duration

we

will

repeat

the

incidental

music

if

it

is

needed

and

then

fade

out

the

music

at

the

end

Event

Slide

List

Music

Sub-Clip

Fig.

Music

and

storyboard

synchronization

a

music

sub-slip

based

scheme

that

is

only

the

top

important

events

are

presented

to

match

the

number

of

music

sub-clips

Rendering

After

the

alignment

between

storyboard

and

incidental

music

in

our

system

fifteen

common

transition

effects

such

as

cross-fade

wipe

and

dissolve

are

also

randomly

selected

to

connect

the

event

slides

producing

a

better

rich

presentation

in

final

rendering

In

this

section

we

evaluate

the

performance

of

the

proposed

approach

to

rich

presentation

and

its

key

component

event

clustering

In

the

experiments

we

randomly

select

topics

of

different

types

including

Earthquake

Halloween

Air

Disaster

US

Election

Nobel

Prize

Britney

Spears

David

Beckham

and

Harry

Potter

from

some

hot

news

topics

in

the

end

of

and

beginning

of

Once

the

topic

is

selected

the

topic

name

is

used

as

a

query

and

the

relevant

documents

are

collected

from

CNN

MSNBC

and

BBC

More

details

about

the

selected

topics

and

the

corresponding

documents

are

shown

in

the

Table

which

lists

the

topic

name

the

time

range

of

the

collected

documents

and

the

number

of

documents

and

its

corresponding

events

Table

A

list

of

testing

topics

in

the

rich

presentation

evaluations

No

Topic

Time

#doc

#event

Earthquake

Halloween

Air

Disaster

US

Election

Britney

Spears

2000-2004 1311

Nobel

Prize

David

Beckham

1995-2004 877

Harry

Potter

Total

It

is

noted

that

in

the

table

only

topics

have

labeled

events

while

another

topics

have

not

This

is

because

that

the

labeling

work

of

a

topic

is

very

subjective

and

usually

hard

for

individuals

to

manually

decide

the

event

number

of

a

given

topic

Therefore

we

only

label

the

topics

which

are

easily

to

be

annotated

based

on

the

criterion

in

Topic

Detection

and

Tracking

-LRB-

TDT

-RRB-

project

-LSB-

-RSB-

For

example

Halloween

is

a

topic

which

is

reported

once

a

year

thus

each

year

's

documents

can

be

regarded

as

an

event

as

for

Earthquake

and

Air

Disaster

their

events

lists

could

be

found

from

corresponding

official

websites

In

the

annotation

we

remove

the

events

which

do

not

have

or

have

few

-LRB-

less

than

-RRB-

relevant

documents

and

also

remove

the

documents

not

belonging

to

any

events

After

parsing

the

obtained

documents

for

each

topic

we

usually

can

obtain

images

per

document

in

average

With

further

duplicate

detection

only

images

per

document

are

remained

Moreover

from

each

document

we

could

also

obtain

about

unique

location

entities

and

unique

name

entities

Other

words

except

of

these

entities

are

taken

as

keywords

Fig.

shows

a

real

representation

of

an

example

document

with

extracted

entities

in

the

XML

format

from

which

the

event

clustering

is

performed

Fig.

XML

representation

of

a

document

on

US

Election

with

extracted

entities

Event

Clustering

As

mentioned

above

the

evaluation

of

the

approach

to

event

clustering

is

evaluated

on

three

topics

including

Earthquake

Halloween

and

Air

Disaster

for

which

the

corresponding

event

numbers

are

determined

and

the

documents

are

labeled

using

a

similar

method

in

the

TDT

project

However

in

the

proposed

appraoch

we

actually

do

not

estimate

the

optimal

event

number

but

use

a

much

larger

one

Therefore

in

order

to

better

evaluate

the

performance

of

the

event

clustering

algorithm

and

compare

with

its

counterpart

we

use

the

event

number

in

the

ground

truth

to

initialize

the

cluster

number

in

the

proposed

clustering

algorithm

URL

URL

Abstract

The

US

battleground

state

of

Ohio

has

certified

the

victory

of

President

George

W

Bush

's

in

last

month

's

poll

Abstract

Date

Date

NLPRESULT

LOCATION

entity

Ohio

entity

freq

freq

entity

US

entity

freq

freq

LOCATION

PERSON

entity

Bush

entity

freq

freq

entity

David

Cobb

entity

freq

freq

PERSON

DATE

entity

December

entity

freq

freq

entity

Friday

entity

freq

freq

DATE

KEYWORDS

entity

recount

entity

freq

freq

entity

elect

entity

freq

freq

entity

America

entity

freq

freq

entity

poll

entity

freq

freq

KEYWORDS

NLPRESULT

In

the

experiments

K-means

which

is

another

frequently

used

clustering

algorithm

-LRB-

as

well

in

TDT

-LSB-

-RSB-

-RRB-

is

adopted

to

compare

with

the

proposed

approach

The

comparison

results

of

two

clustering

approaches

are

illustrated

in

Table

with

precision

and

recall

for

each

topic

Table

The

performance

comparison

between

our

approach

and

K-means

on

the

event

clustering

Precision

Recall

K-means

Ours

K-means

Ours

Earthquake

Halloween

Air

Disaster

Average

From

Table

it

can

be

seen

that

the

results

of

our

approach

are

significantly

better

than

those

of

K-means

both

on

precision

and

recall

On

the

three

testing

topics

the

average

precision

of

our

approach

is

up

to

and

the

average

recall

achieves

which

is

and

higher

than

those

of

K-means

respectively

By

tracing

the

process

of

K-means

we

find

that

K-means

usually

assigns

documents

far

away

from

each

other

on

the

timeline

into

the

same

cluster

since

the

time

information

affects

little

in

K-means

It

also

indicates

the

advantages

of

our

approach

with

time

modeling

The

algorithms

also

show

different

performance

on

different

kind

topics

As

for

the

Air

disaster

its

performance

is

not

as

good

as

that

of

the

other

two

since

the

features

-LRB-

words

and

time

-RRB-

of

its

events

are

more

complicated

and

intertwined

in

the

feature

space

As

for

the

topics

-LRB-

in

Table

I

-RRB-

which

could

not

have

an

objective

evaluation

the

clustering

performance

on

these

topics

could

be

indirectly

reflected

by

the

subjective

evaluation

of

the

rich

presentation

presented

in

section

This

is

because

users

will

be

more

satisfied

when

the

grouped

documents

shown

in

each

event

slide

really

belong

to

the

same

event

while

users

are

not

satisfied

if

the

documents

from

different

events

are

mixed

in

one

event

slide

Rich

Presentation

It

is

usually

difficult

to

find

a

quantitative

measure

for

rich

presentation

since

the

assessment

of

the

goodness

of

rich

presentation

is

a

strong

subjective

task

In

this

paper

we

carry

out

a

preliminary

user

study

to

evaluate

the

performance

of

the

proposed

rich

presentation

schemes

To

indicate

the

performance

of

rich

presentation

we

design

two

measures

in

the

experiments

including

informativeness

and

enjoyablity

following

the

criteria

used

in

the

work

-LSB-

-RSB-

Here

the

informativeness

measures

whether

the

subjects

satisfy

with

the

information

obtained

from

the

rich

presentation

while

enjoyablity

indicates

if

users

feel

comfortable

and

enjoyable

when

they

are

reading

the

rich

presentation

In

evaluating

the

informativeness

we

also

provide

the

documents

from

which

the

rich

presentation

is

generated

They

are

used

as

baseline

based

on

which

the

subjects

can

more

easily

evaluate

if

the

important

overview

information

contained

in

the

documents

is

conveyed

by

the

rich

presentation

Moreover

in

order

to

reveal

the

subjects

opinion

on

the

design

of

the

storyboard

template

like

the

one

shown

in

Fig

we

also

ask

the

subjects

to

evaluate

the

interface

design

In

the

user

study

volunteered

subjects

including

males

and

females

are

invited

The

subjects

are

around

years

old

have

much

experience

on

computer

manipulation

and

usually

read

news

on

web

in

their

leisure

time

We

ask

them

to

give

a

subjective

score

between

and

for

each

measure

of

the

rich

presentation

of

each

testing

topic

-LRB-

an

exception

is

interface

design

which

is

the

same

for

each

rich

presentation

-RRB-

Here

the

score

to

stands

for

unsatisfied

-LRB-

-RRB-

somewhat

unsatisfied

-LRB-

-RRB-

acceptable

-LRB-

-RRB-

satisfied

-LRB-

-RRB-

and

very

satisfied

-LRB-

-RRB-

respectively

In

experiments

we

first

check

with

the

interface

design

measure

We

find

out

of

subjects

satisfy

with

the

event

template

design

and

the

left

three

also

think

it

is

acceptable

The

average

score

is

up

to

An

interesting

observation

is

that

some

subjects

like

the

template

design

very

much

at

the

first

glance

but

they

feel

a

little

boring

after

they

finish

all

the

user

study

since

every

slide

in

the

rich

presentation

of

each

topic

has

the

same

appearance

It

hints

us

that

we

had

better

design

different

templates

for

different

topics

to

make

the

rich

presentation

more

attractive

As

for

the

other

two

measures

we

average

the

score

across

all

the

subjects

to

represent

the

performance

for

each

topic

and

list

the

detailed

results

in

Table

It

can

be

seen

that

the

average

score

of

both

enjoyablity

and

informativeness

achieves

which

indicates

that

most

subjects

satisfy

the

provided

overview

information

of

the

target

topic

and

they

enjoy

themselves

when

reading

these

rich

presentations

Table

The

evaluation

results

of

rich

presentation

on

each

topic

No

Topic

Informative

Enjoyable

Earthquake

Halloween

Air

Disaster

US

Election

Britney

Spears

Nobel

Prize

David

Beckham

Harry

Potter

Average

In

the

experiments

we

find

informativeness

is

highly

depended

on

the

correlation

between

the

presented

documents

and

the

target

topic

If

the

presented

information

is

consistent

with

the

topic

subjects

usually

give

a

high

score

for

informativeness

such

as

those

on

Earthquake

and

US

Election

otherwise

they

will

give

a

low

score

like

those

on

David

Beckham

and

Nobel

Prize

It

indicates

that

it

is

quite

important

to

provide

users

clean

information

of

the

target

topic

with

less

noise

However

in

current

system

the

documents

are

crawled

from

web

and

inevitably

contain

many

noises

It

affects

much

on

the

performance

of

informativeness

in

the

current

system

We

need

to

consider

how

to

prone

the

information

of

the

target

topic

in

the

future

works

We

also

find

that

the

enjoyablity

score

is

usually

related

with

informativeness

If

the

subjects

do

not

get

enough

information

from

the

rich

presentation

they

will

be

not

enjoyable

as

well

such

as

the

topics

of

Nobel

Prize

and

Harry

Potter

Enjoyablity

is

also

topic-related

the

subjects

usually

feel

unconformable

when

they

are

facing

with

miserable

topics

such

as

Earthquake

and

Air

Disaster

although

their

informativeness

is

quite

high

On

the

contrary

users

give

a

high

score

for

enjoyablity

on

the

interesting

topics

such

as

Britney

Spears

and

David

Beckham

although

their

informative

score

is

not

high

This

is

because

that

there

are

usually

many

funny

and

interesting

pictures

in

the

presentation

of

these

topics

Another

finding

is

that

users

usually

fell

unenjoyable

if

the

images

and

summaries

in

one

event

slide

are

not

consistent

with

each

other

From

this

view

the

high

enjoyablity

score

in

our

experiments

also

indicates

that

our

event

clustering

algorithm

works

promisingly

To

facilitate

users

to

quickly

grasp

and

go

through

the

content

of

a

semantic

topic

in

this

paper

we

have

proposed

a

novel

approach

to

rich

presentation

to

generate

a

concise

and

informative

storyboard

for

the

target

topic

with

many

relevant

multimodal

information

including

image

text

audio

and

video

In

this

approach

the

related

multimodal

information

of

a

given

topic

is

first

extracted

from

news

databases

Then

the

events

are

clustered

and

the

corresponding

information

such

as

representative

images

geographic

information

and

event

summary

is

obtained

The

information

is

composed

into

an

attractive

storyboard

which

is

finally

synchronized

with

incidental

music

A

user

study

indicates

that

the

presented

system

works

well

on

our

testing

examples

There

is

still

some

room

for

improving

the

proposed

approach

First

the

proposed

approach

could

be

extended

to

other

multimedia

databases

or

more

general

websites

For

example

some

standard

multimedia

database

like

NIST

TRECVID

could

provide

a

nice

platform

for

the

implementation

and

evaluation

of

event

detection

and

rich

presentation

Second

to

integrate

more

relevant

multimedia

information

-LRB-

such

as

video

clips

and

flashes

-RRB-

and

more

accurate

information

regarding

the

target

topic

is

highly

expected

by

users

Thus

more

advanced

information

retrieval

extraction

techniques

and

other

multimedia

analysis

techniques

are

needed

to

be

exploited

and

integrated

such

as

relevance

ranking

mapping

schemes

important

or

representative

video

clips

detection

and

video

clip

summarization

We

also

need

to

design

a

much

natural

way

to

incorporate

video

clips

in

the

event

template

Third

we

also

consider

designing

various

storyboard

templates

for

different

kind

of

topics

For

example

each

topic

may

be

belonging

to

different

clusters

such

as

politics

sports

and

entertainments

each

of

which

can

have

a

representative

template

Forth

appropriate

user

interaction

will

be

added

to

further

make

the

storyboard

more

interactive

and

easy

to

control

Finally

a

thorough

evaluation

will

be

implemented

to

evaluate

the

effect

of

each

component

in

the

framework

and

storyboard

template

-LSB-

-RSB-

A.

Vailaya

M.A.T.

Figueiredo

A.

K.

Jain

and

H.-J

Zhang

Image

classification

for

content-based

indexing

IEEE

Transactions

on

Image

Processing

Vol

Iss

-LSB-

-RSB-

F.

J.

M.-J

Li

H.-J

Zhang

and

B.

Zhang

An

effective

region-based

image

retrieval

framework

Proc

ACM

Multimedia

pp.

-LSB-

-RSB-

J.

Platt

AutoAlbum

Clustering

Digital

Photographs

using

Probabilistic

Model

Merging

Proc

IEEE

Workshop

on

Content-Based

Access

of

Image

and

Video

Libraries

pp.

-LSB-

-RSB-

A.

Hanjalic

R.

L.

Lagendijk

J.

Biemond

Automated

high-level

movie

segmentation

for

advanced

video-retrieval

systems

IEEE

Trans

on

Circuits

and

Systems

For

Video

Technology

Vol

No.

pp.

-LSB-

-RSB-

J.

Assfalg

and

et

al

Semantic

annotation

of

soccer

videos

automatic

highlights

identification

CVIU

vol

pp

-LSB-

-RSB-

A.

Ekin

A.

M.

Tekalp

and

R.

Mehrotra

Automatic

soccer

video

analysis

and

summarization

IEEE

Trans

on

Image

Processing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

Y.

F.

Ma

L.

Lu

H.

J.

Zhang

and

M.-J

Li

A

User

Attention

Model

for

Video

Summarization

ACM

Multimeida

pp.

-LSB-

-RSB-

L.

Xie

P.

Xu

S.F.

Chang

A.

Divakaran

and

H.

Sun

Structure

analysis

of

soccer

video

with

domain

knowledge

and

hidden

markov

models

Pattern

Recognition

Letters

vol

-LRB-

-RRB-

pp.

-LSB-

-RSB-

L.

Lu

H.

Jiang

H.

J.

Zhang

A

Robust

Audio

Classification

and

Segmentation

Method

Proc

ACM

Multimedia

pp

-LSB-

-RSB-

R.

Cai

L.

Lu

H.-J

Zhang

and

L.-H

Cai

Highlight

Sound

Effects

Detection

in

Audio

Stream

Proc

ICME

Vol

-LSB-

-RSB-

Y.

Rui

A.

Gupta

and

A.

Acero

Automatically

Extracting

Highlights

for

TV

Baseball

Programs

Proc

ACM

Multi-media

-LSB-

-RSB-

C.

Snoek

and

M.

Worring

Multimodal

Video

Indexing

A

Review

of

the

State-of-the-art

Multimedia

Tools

and

Applications

Vol

No.

pp.

-LSB-

-RSB-

E.M.

Voorhees

Query

expansion

using

lexical-semantic

relations

Proc

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

pp

-LSB-

-RSB-

Z.-W

Li

M.-J

Li

and

W.-Y

Ma

A

Probabilistic

Model

for

Retrospective

News

Event

Detection

Proc

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

-LSB-

-RSB-

D.

M.

Bikel

R.

L.

Schwartz

and

R.

M.

Weischedel

An

Algorithm

That

Learns

What

's

in

a

Name

Machine

Learning

-LRB-

-RRB-

-LSB-

-RSB-

K.

Nigam

A.

McCallum

S.

Thrun

and

T.

Mitchell

Text

Classification

from

Labeled

and

Unlabeled

Documents

using

EM

Machine

Learning

-LRB-

-RRB-

-LSB-

-RSB-

T.

Hastie

R.

Tibshirani

and

J.

Friedman

The

Elements

of

Statistical

Learning

Data

Mining

Inference

and

Prediction

Springer-Verlag

-LSB-

-RSB-

MapPoint

Web

Service

products

webservice/default

mspx

-LSB-

-RSB-

X.-S

Hua

L.

Lu

H.-J

Zhang

Automated

Home

Video

Editing

Proc

ACM

Multimedia

pp.

-LSB-

-RSB-

J.

Foote

M.

Cooper

and

A.

Girgensohn

Creating

Music

Videos

Using

Automatic

Media

Analysis

ACM

Multimedia

-LSB-

-RSB-

Topic

Detection

and

Tracking

-LRB-

TDT

-RRB-

Project

nist.gov

speech/tests/tdt

-LSB-

-RSB-

J.

Allan

R.

Papka

and

V.

Lavrenko

On-line

New

Event

Detection

and

Tracking

Proc

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

storyboard
representative media
multimedia authoring
rich presentation
event clustering
events clustering

documentary and movie
subjective multiple events
rich video clips and flashes
multimodality
multi-modal information
high-level semantics
generate storyboard
multimedia fusion
communication and multimedia
images, videos and audio technologies