Topic

Transition

Detection

Using

Hierarchical

Hidden

Markov

and

Semi-Markov

Models

In

this

paper

we

introduce

a

probabilistic

framework

to

exploit

hierarchy

structure

sharing

and

duration

information

for

topic

transition

detection

in

videos

Our

probabilistic

detection

framework

is

a

combination

of

a

shot

classification

step

and

a

detection

phase

using

hierarchical

probabilistic

models

We

consider

two

models

in

this

paper

the

extended

Hierarchical

Hidden

Markov

Model

-LRB-

HHMM

-RRB-

and

the

Coxian

Switching

Hidden

semi-Markov

Model

-LRB-

S-HSMM

-RRB-

because

they

allow

the

natural

decomposition

of

semantics

in

videos

including

shared

structures

to

be

modeled

directly

and

thus

enabling

efficient

inference

and

reducing

the

sample

complexity

in

learning

Additionally

the

S-HSMM

allows

the

duration

information

to

be

incorporated

consequently

the

modeling

of

long-term

dependencies

in

videos

is

enriched

through

both

hierarchical

and

duration

modeling

Furthermore

the

use

of

the

Coxian

distribution

in

the

S-HSMM

makes

it

tractable

to

deal

with

long

sequences

in

video

Our

experimentation

of

the

proposed

framework

on

twelve

educational

and

training

videos

shows

that

both

models

outperform

the

baseline

cases

-LRB-

flat

HMM

and

HSMM

-RRB-

and

performances

reported

in

earlier

work

in

topic

detection

The

superior

performance

of

the

S-HSMM

over

the

HHMM

verifies

our

belief

that

duration

information

is

an

important

factor

in

video

content

modeling

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Content

Analysis

and

Indexing

Algorithms

Management

The

ultimate

goal

of

the

video

segmentation

problem

is

to

characterize

the

temporal

dynamics

of

the

video

whereby

it

can

be

segmented

into

coherent

units

possibly

at

different

levels

of

abstraction

Seeking

abstract

units

to

move

beyond

the

shots

has

thus

been

an

active

topic

of

much

recent

re-Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

MM

November

Singapore

Copyright

ACM

search

While

the

problem

of

shot

transition

is

largely

solved

at

a

satisfactory

level

-LSB-

-RSB-

the

abstract

units

or

scene

detection

problem

is

much

harder

partially

due

to

the

following

three

challenges

identified

in

-LSB-

-RSB-

-LRB-

a

-RRB-

the

variety

in

directional

styles

-LRB-

b

-RRB-

the

semantic

relationship

of

neighbouring

scenes

and

-LRB-

c

-RRB-

the

knowledge

of

the

viewer

about

the

world

While

the

last

aspect

is

beyond

the

scope

of

this

work

the

first

two

clearly

imply

that

effective

modeling

of

high-level

semantics

requires

the

domain

knowledge

-LRB-

directional

style

-RRB-

and

the

modeling

of

long-term

multiple-scale

correlations

of

the

video

dynamics

-LRB-

neighboring

semantic

relationship

-RRB-

Modeling

temporal

correlations

over

a

long

period

is

generally

a

challenging

problem

As

we

shall

review

in

the

subsequent

section

this

problem

is

usually

solved

in

a

specific

domain

setting

so

that

the

expert

knowledge

about

the

domain

can

be

utilised

While

organization

of

content

in

generic

videos

-LRB-

e.g.

movies

-RRB-

is

too

diverse

to

be

fully

char-acterized

by

statistical

models

the

hierarchy

of

semantic

structure

in

the

class

of

education-oriented

videos

is

more

defined

exposing

strong

temporal

correlation

in

time

and

thus

make

it

more

desirable

to

probabilistic

modeling

In

this

paper

we

concentrate

on

this

video

genre

and

develop

an

effective

framework

to

segment

these

videos

into

topi-cally

correlated

units

This

problem

is

an

important

step

to

enable

abstraction

summarization

and

browsing

of

educational

content

a

rich

class

of

film

genre

that

has

an

increasing

important

role

in

building

e-services

for

learning

and

training

Probabilistic

modeling

of

temporal

correlations

in

video

data

is

however

a

difficult

problem

It

is

complicated

because

the

underlying

semantics

naturally

possess

a

hierarchical

decomposition

with

possible

existence

of

tight

structure

sharing

between

high-level

semantics

In

addition

the

typical

duration

for

these

structures

usually

varies

for

each

of

its

higher

semantic

As

an

example

assisted

narration

a

section

that

involves

the

narrator

talking

to

the

audience

is

usually

used

in

both

the

introduction

and

in

the

main

body

of

a

topic

in

an

educational

video

However

while

one

or

rarely

two

shots

of

assisted

narration

-LRB-

AN

-RRB-

are

considered

sufficient

for

the

introduction

the

body

typically

requires

many

AN

shots

Thus

it

is

important

to

exploit

and

fuse

hierarchical

decomposition

structure

sharing

and

duration

information

in

a

unified

framework

to

effectively

address

the

problem

of

topic

transition

detection

The

most

widely

used

pobabilistic

model

is

the

hidden

Markov

model

-LRB-

HMM

-RRB-

However

in

many

cases

the

HMM

is

unsuitable

for

video

analysis

since

the

strong

Markov

assumption

makes

it

too

restrictive

to

capture

correlations

over

long

periods

This

limitation

is

usually

overcome

in

the

literature

by

the

use

of

a

series

of

HMMs

in

a

hierarchic

manner

The

underlying

problem

in

these

approaches

still

is

the

manual

combination

of

HMMs

at

the

higher

levels

which

results

in

the

excessive

expense

of

preparing

the

training

data

and

more

importantly

the

interaction

across

higher

semantic

levels

is

not

incorporated

during

model

training

One

rigorous

approach

to

overcome

this

limitation

is

the

use

of

the

Hierarchical

Hidden

Markov

Model

-LRB-

HHMM

-RRB-

first

introduced

in

-LSB-

-RSB-

and

later

extended

to

handle

structure

sharing

in

-LSB-

-RSB-

The

sophisticated

model

in

-LSB-

-RSB-

allows

natural

hierarchical

organization

of

the

videos

including

any

existing

structure

sharing

to

be

modeled

rigorously

Practically

this

will

result

in

computational

savings

and

a

reduction

in

sample

complexity

for

learning

Given

its

advantages

we

use

this

model

in

this

paper

to

model

educational

video

content

for

topic

transition

detection

It

is

natural

to

see

that

durative

properties

play

an

important

role

in

human

perception

An

excessively

long

lecture

would

bore

the

students

As

such

education-oriented

videos

-LRB-

e.g.

news

documentaries

lectures

training

videos

etc.

-RRB-

exhibit

strong

duration

information

in

their

content

We

thus

propose

an

alternative

approach

towards

handling

temporal

dependencies

over

long

periods

through

the

explicit

modeling

of

duration

information

captured

in

semi-Markov

models

In

these

models

a

state

is

assumed

to

remain

un-changed

for

some

duration

of

time

before

it

transits

to

a

new

state

and

thus

it

addresses

the

violation

of

the

strong

Markov

assumption

from

having

states

whose

duration

distributions

are

non-geometric

Existing

semi-Markov

models

commonly

model

duration

distributions

as

multinomials

Video

data

is

however

typically

very

long

thus

making

a

multinomial

semi-Markov

model

unsuitable

for

video

analysis

since

it

would

result

in

both

excessive

computation

and

the

number

of

parameters

required

Continuous

modeling

of

duration

does

exist

such

as

in

the

use

of

the

Gamma

distribution

or

more

generally

the

exponential

family

described

in

-LSB-

-RSB-

to

provide

more

compact

parameterization

However

there

are

still

two

limitations

applied

to

these

models

for

video

analysis

-LRB-

a

-RRB-

learning

these

distributions

requires

numerical

optimiza-tion

and

the

time

complexity

still

depends

on

the

maximum

duration

length

and

-LRB-

b

-RRB-

no

hierarchical

modeling

has

been

attempted

Fortunately

in

-LSB-

-RSB-

a

Switching

Hidden

Semi-Markov

Model

-LRB-

S-HSMM

-RRB-

is

introduced

in

which

the

duration

is

modeled

as

a

discrete

M

phase

Coxian

distribution

This

model

is

particularly

interesting

for

video

analysis

since

-LRB-

-RRB-

it

can

model

hierarchical

decomposition

and

-LRB-

-RRB-

the

Coxian

duration

modeling

results

in

fast

learning

and

inference

the

number

of

parameters

is

small

and

close-formed

estimation

exists

Parameterizing

long-term

temporal

correlations

existing

in

video

is

thus

enriched

by

both

the

hierarchical

architecture

and

the

duration

modeling

at

the

bottom

level

of

the

S-HSMM

To

model

video

content

we

argue

that

it

is

beneficial

to

exploit

both

the

hierarchical

organization

of

the

videos

their

semantically

shared

substructures

and

typical

durations

of

important

semantics

These

aspects

are

all

addressed

in

this

paper

in

a

unified

and

coherent

probabilistic

framework

We

use

the

HHMM

and

the

S-HSMM

and

propose

a

two-phase

architecture

for

detecting

topical

transition

in

educational

videos

In

the

first

phase

shots

are

classified

into

meaningful

labels

Using

classified

shot

labels

the

second

phase

trains

a

hierarchical

probabilistic

model

-LRB-

HHMM

or

S-HSMM

-RRB-

which

is

then

used

at

a

later

stage

for

segmentation

and

annotation

Prior

knowledge

about

the

domain

including

shared

structures

is

incorporated

into

the

topo-logical

structure

during

training

Our

cross-validation

on

a

dataset

including

a

mix

of

twelve

videos

demonstrates

promising

results

The

performances

from

the

baseline

cases

-LRB-

HMM

and

HSMM

-RRB-

have

shown

that

they

are

too

restrictive

and

unsuitable

in

our

detection

scheme

proving

the

validity

of

hierarchical

modeling

The

performances

of

the

hierarchical

models

including

the

HHMM

and

S-HSMM

are

shown

to

surpass

all

results

reported

in

earlier

work

in

topic

detection

-LSB-

-RSB-

The

superior

performance

of

the

S-HSMM

over

the

HHMM

has

also

demonstrated

our

belief

that

duration

information

is

indeed

an

important

element

in

the

segmentation

problem

Exploiting

the

hierarchy

structure

sharing

and

duration

in

a

unified

probabilistic

framework

our

contributions

are

twofold

-LRB-

-RRB-

we

provide

a

coherent

hierarchical

probabilistic

framework

for

topic

detection

Although

the

current

report

concentrates

on

the

educational

genre

this

framework

can

clearly

generalize

to

other

genres

such

as

news

and

documentaries

and

-LRB-

-RRB-

to

our

knowledge

we

are

the

first

to

investigate

duration

and

hierarchical

modeling

for

video

segmentation

in

a

unified

framework

The

remainder

of

this

paper

is

organized

as

follows

In

the

next

section

we

provide

related

background

to

this

work

This

is

followed

by

a

detailed

section

on

the

detection

framework

including

the

description

of

the

HHMM

and

S-HSMM

We

detail

the

shot

classification

phase

in

Section

Experimental

results

are

then

reported

in

Section

Finally

the

conclusion

follows

in

Section

Seeking

high-level

semantics

to

move

beyond

the

shots

has

been

the

central

theme

of

much

recent

research

Attempts

towards

this

problem

have

resulted

in

a

fast

growing

body

of

work

and

depending

on

the

investigating

domain

the

abstracting

units

appear

under

different

names

such

as

scene

story

episode

for

motion

pictures

topic

subtopic

macro

segments

story

units

for

information-oriented

videos

-LRB-

news

documentaries

training

and

educational

videos

-RRB-

or

general

term

like

logical

story

units

used

in

-LSB-

-RSB-

Otherwise

stated

we

shall

the

term

scene

in

this

section

to

mean

all

of

the

aforementioned

names

Early

attempts

have

targeted

extracting

scene-level

concepts

in

broadcast

programs

in

particular

news

videos

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

In

these

attempts

the

semantic

extraction

problem

is

usually

cast

as

the

classification

problem

The

authors

in

-LSB-

-RSB-

for

example

combine

a

number

of

visual

and

aural

low-level

features

together

with

shot

syntax

presented

in

news

videos

to

group

shots

into

different

narrative

structures

and

label

them

as

anchor-shot

voice-over

or

Since

topic

change

coincides

with

a

shot

transition

the

shot

boundary

provides

crucial

information

in

detecting

topic

transitions

therefore

the

term

duration

in

this

work

is

calculated

in

terms

of

the

number

of

shots

This

drastically

simplifies

the

modeling

process

An

alternative

way

of

modeling

duration

is

to

uniformly

replicate

a

shot

label

based

on

its

length

However

doing

this

would

require

an

extra

modeling

of

shot

transition

knowledge

In

this

work

we

avoid

this

complication

and

concentrate

on

duration

information

based

on

the

shot

counts

view

Liu

et

al.

-LSB-

-RSB-

propose

a

video/audio

fusion

approach

to

segment

news

reports

from

other

categories

in

broadcast

programs

with

different

types

of

classifiers

-LRB-

simple

threshold

method

Gaussian

mixture

classifier

and

support

vector

machine

-RRB-

Ide

et

al.

-LSB-

-RSB-

propose

an

automatic

indexing

scheme

for

news

video

where

shots

are

indexed

based

on

the

image

content

and

keywords

into

five

categories

speech/report

anchor

walking

gathering

and

computer

graphics

Caption

text

information

is

then

used

with

classified

shots

to

build

the

indices

Segmentation

of

the

news

story

is

the

second

major

theme

explored

in

the

broadcast

domain

The

common

underlying

approach

used

in

these

works

is

the

use

of

explicit

rules

about

the

structure

of

news

programs

to

locate

the

transitions

of

a

news

story

Commonly

accepted

heuristics

are

for

example

a

news

story

often

starts

and

finishes

with

anchor-person

shots

-LSB-

-RSB-

the

start

of

a

news

story

is

usually

coupled

with

music

-LSB-

-RSB-

or

a

relative

long

silence

period

is

the

indication

of

the

boundary

between

two

news

stories

-LSB-

-RSB-

More

complicated

rules

via

temporal

analysis

are

also

exploited

such

as

the

work

of

-LSB-

-RSB-

which

utilises

detection

results

of

anchor-persons

and

captions

to

form

a

richer

set

of

rules

-LRB-

i.e.

if

the

same

text

caption

appears

in

two

consecutive

anchor-person

shots

then

they

belong

to

the

same

news

story

-RRB-

There

is

also

a

body

of

work

which

casts

the

segmentation

problem

of

news

story

in

a

HMM

framework

-LSB-

-RSB-

The

authors

in

-LSB-

-RSB-

for

example

propose

the

news

segmentation

problem

as

problem

of

decoding

the

maximum

state

sequence

of

a

trained

HMM

whose

transition

matrix

is

tailored

by

explicit

rules

about

the

news

program

A

somewhat

similar

approach

to

the

work

in

this

paper

is

-LSB-

-RSB-

-LRB-

whose

results

came

first

in

the

story

segmentation

benchmark

-RRB-

Shots

in

-LSB-

-RSB-

are

first

classified

into

a

set

common

labels

in

news

-LRB-

e.g.

anchor

text-scene

etc.

-RRB-

These

labels

are

then

input

to

a

HMM

for

the

segmentation

task

They

report

best

performances

of

recall

and

precision

for

the

TRECVID

dataset

The

work

of

-LSB-

-RSB-

however

remains

limited

due

to

the

flat

structure

HMM

and

it

is

not

clear

how

the

set

of

transition

states

were

chosen

In

an

effort

to

move

beyond

flat

structure

the

authors

of

-LSB-

-RSB-

have

raised

the

need

for

high-order

statistical

techniques

which

will

be

addressed

in

this

paper

through

the

HHMM

and

S-HSMM

More

recent

approaches

towards

scene

extraction

have

shifted

to

motion

pictures

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Detecting

scenes

in

motion

pictures

is

in

general

a

challenging

problem

and

there

are

three

main

existing

approaches

as

outlined

in

-LSB-

-RSB-

temporal

clustering-based

rule-based

and

memory-based

detection

In

the

clustering-based

approach

shots

are

grouped

into

scenes

based

on

visual

similarity

and

temporal

closeness

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Scene

breaks

in

the

rule-based

detection

approach

are

determined

based

on

the

semantic

and

syntactic

analysis

of

audiovisual

characteristics

and

in

some

cases

further

enhanced

with

more

rigorous

grammars

from

film

theory

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

The

authors

in

-LSB-

-RSB-

propose

a

memory-based

scene

detection

framework

Visual

shot

similarity

in

these

works

is

determined

based

on

the

consistency

in

color

chromaticality

and

the

soundtrack

is

partitioned

into

audio

scenes

Visual

and

aural

data

are

then

fused

within

a

framework

of

memory

and

attention

span

model

to

find

likely

scene

breaks

or

singleton

events

Further

related

background

on

scene

detection

can

be

found

in

many

good

surveys

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Existing

HMM-based

approaches

towards

modeling

long-term

temporal

dependencies

typically

use

pre-segmented

training

data

at

multiple

levels

and

hierarchically

train

a

pool

of

HMMs

in

which

HMMs

at

the

lower

levels

are

used

as

input

to

the

HMMs

at

the

upper

levels

In

principle

some

fundamental

units

are

recognised

by

a

sequence

of

HMMs

and

then

likelihood

values

-LRB-

or

labels

-RRB-

obtained

from

these

HMMs

are

combined

to

form

a

hierarchy

of

HMMs

to

capture

the

interactions

at

higher

semantic

levels

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Analysing

sports

videos

Kijak

et

al.

-LSB-

-RSB-

propose

a

two-tiered

classification

of

tennis

videos

using

two

layers

of

HMMs

At

the

bottom

level

four

HMMs

are

used

to

model

four

shot

classes

-LRB-

first

missed

serve

rally

replay

and

break

-RRB-

Each

HMM

is

trained

separately

and

subse-quently

topped

up

by

another

HMM

at

the

top

level

which

represents

the

syntax

of

the

tennis

video

with

three

states

of

the

game

sets

games

and

points

Parameters

for

the

top

HMM

are

however

all

manually

specified

In

-LSB-

-RSB-

a

generic

two-level

hierarchy

of

HMMs

is

proposed

to

detect

recurrent

events

in

movies

and

talk

shows

Their

idea

is

to

use

an

ergodic

HMM

at

the

top

level

in

which

each

state

is

another

-LRB-

non-ergodic

-RRB-

sub-HMM

representing

a

type

of

signal

stationary

properties

For

the

case

of

movies

the

top

HMM

has

six

states

and

each

is

in

turn

another

three-state

non-ergodic

HMM

The

observations

are

modelled

as

a

mixture

of

Gaussians

After

training

the

authors

claim

that

interesting

events

can

be

detected

such

as

explosion

male

speech

and

so

on

While

being

able

to

overcome

the

limitation

of

the

flat

HMM

in

modeling

long-term

dependencies

approaches

that

use

HMMs

at

multiple

levels

still

suffer

from

two

major

problems

-LRB-

-RRB-

pre-segmented

and

an-notated

data

are

needed

at

all

levels

for

training

and

-LRB-

-RRB-

in

most

existing

work

parameterization

at

higher

levels

has

to

be

manually

specified

In

many

cases

preparing

training

data

at

multiple

levels

is

extremely

tedious

and

at

worst

may

not

be

possible

With

respect

to

the

second

problem

since

each

semantic

level

has

to

be

modeled

separately

the

underlying

problem

is

that

the

interactions

across

semantic

layers

are

not

modeled

and

thus

do

not

contribute

to

the

learning

process

One

framework

that

integrates

the

semantics

across

layers

is

the

Hierarchical

Hidden

Markov

Model

-LRB-

HHMM

-RRB-

proposed

recently

in

-LSB-

-RSB-

The

hierarchical

HMM

extends

the

standard

HMM

in

a

hierarchic

manner

to

allow

each

state

to

be

recursively

generalised

as

another

sub-HMM

and

thus

enabling

the

ability

to

handle

hierarchical

modeling

of

complex

dynamic

processes

in

particular

the

ability

to

infer

correlated

observations

over

long

periods

in

the

observation

sequence

via

the

higher

levels

of

hierarchy

-LSB-

-RSB-

The

original

motivation

in

-LSB-

-RSB-

was

to

seek

better

modeling

of

different

stochastic

levels

and

length

scales

presented

in

language

-LRB-

e.g.

speech

handwriting

or

text

-RRB-

However

the

model

introduced

in

-LSB-

-RSB-

considers

only

state

hierarchies

that

have

tree

structures

disallowing

the

sharing

of

substructures

among

the

high-level

states

Recognizing

this

need

the

authors

in

-LSB-

-RSB-

have

extended

the

strict

tree-form

topology

in

the

original

HHMMs

of

-LSB-

-RSB-

and

allowed

it

to

be

a

general

lattice

structure

The

extension

thus

permits

a

state

at

any

arbitrary

level

of

the

HHMMs

to

be

shared

by

more

than

one

parental

state

at

its

higher

level

-LRB-

i.e.

resulting

in

a

compact

form

of

parameter

typing

at

multiple

levels

-RRB-

This

extended

Not

to

be

confused

with

the

Hierarchical

HMMs

form

is

very

attractive

for

video

content

modeling

since

it

allows

the

natural

organization

of

the

video

content

to

be

modeled

not

only

in

terms

of

multiple

scales

but

also

in

terms

of

shared

substructures

existing

in

the

decomposition

Further

details

on

the

HHMM

are

provided

in

Section

Early

application

of

the

HHMM

for

video

analysis

is

found

in

-LSB-

-RSB-

and

later

extended

in

-LSB-

-RSB-

In

these

works

the

authors

use

the

HHMM

to

detect

the

events

of

play

and

break

in

soccer

videos

For

inference

and

learning

the

HHMM

is

collapsed

into

a

flat

HMM

with

a

very

large

product

state

space

which

can

then

be

used

in

conjunction

with

the

standard

forward/backward

passes

as

in

a

normal

HMM

Four

methods

are

compared

in

-LSB-

-RSB-

to

detect

play

and

break

-LRB-

-RRB-

supervised

HMMs

in

which

each

category

is

trained

with

a

separate

HMM

-LRB-

-RRB-

supervised

HHMMs

in

which

bottom

level

HMMs

are

learned

separately

and

parameters

for

the

upper

levels

are

manually

specified

-LRB-

-RRB-

unsupervised

HHMMs

without

model

adaptation

and

-LRB-

-RRB-

supervised

HHMMs

with

model

adaptation

In

-LRB-

-RRB-

and

-LRB-

-RRB-

two-level

HHMMs

are

used

Their

results

have

shown

a

very

close

match

between

unsupervised

and

supervised

methods

in

which

the

completely

unsupervised

method

with

model

adaptation

performs

marginally

better

These

figures

are

and

respectively

for

those

four

methods

While

presenting

a

novel

contribution

to

the

feature

selection

and

model

selection

procedure

the

application

of

the

HHMMs

in

this

work

is

still

limited

both

for

learning

and

for

exploitation

of

the

hierarchical

structure

Flattening

a

HHMM

into

a

flat

HMM

as

done

in

-LSB-

-RSB-

suffers

from

many

drawbacks

as

criticized

in

-LSB-

-RSB-

-LRB-

a

-RRB-

it

can

not

provide

multi-scale

interpretation

-LRB-

b

-RRB-

it

loses

modularity

since

the

parameters

for

the

flat

HMM

get

constructed

in

a

complex

manner

and

-LRB-

c

-RRB-

it

may

introduce

more

parameters

and

most

importantly

it

does

not

have

the

ability

to

reuse

parameters

in

other

words

parameters

for

the

shared

sub-models

are

not

tied

during

the

learning

but

have

to

be

replicated

and

thus

lose

the

inherent

strength

of

hierarchical

modeling

Being

able

to

model

shared

structures

the

extended

HHMMs

of

-LSB-

-RSB-

allows

us

to

build

more

compact

models

which

facilitates

more

efficient

inference

and

reduces

the

sample

complexity

in

learning

This

model

is

applied

in

-LSB-

-RSB-

and

-LSB-

-RSB-

for

the

problem

of

topic

transition

detection

and

video

structure

discovery

respectively

The

authors

in

-LSB-

-RSB-

use

a

three-level

HHMM

for

the

detection

of

topic

transitions

in

educational

videos

Differing

from

our

experiments

in

this

paper

the

HHMM

in

-LSB-

-RSB-

is

modified

to

operate

directly

with

continuous-valued

observed

data

via

the

use

of

Gaussian

mixture

models

as

the

emission

probabilities

Each

shot-based

observed

vector

consists

of

seven

features

extracted

from

visual

and

audio

streams

They

report

a

recall

rate

and

precision

for

the

detection

task

In

another

application

with

the

help

of

prior

knowledge

about

educational

videos

a

topology

for

a

three-level

HHMM

is

used

in

-LSB-

-RSB-

to

automatically

discover

meaningful

narrative

units

in

the

educational

genre

Their

experiments

have

shown

encouraging

results

in

which

many

meaningful

structures

are

hierarchically

discovered

such

as

on-screen

narration

with

texts

expressive

linkage

expressive

voice-over

etc.

The

work

of

-LSB-

-RSB-

is

somewhat

similar

to

that

of

-LSB-

-RSB-

reviewed

earlier

in

this

section

except

the

model

in

-LSB-

-RSB-

allows

more

domain

knowledge

to

be

encoded

and

the

parameters

are

all

learned

automatically

Our

topic

detection

framework

consists

of

two

phases

The

first

phase

performs

shot

detection

and

low

level

feature

extraction

and

then

classifies

a

shot

in

a

meaningful

label

set

This

phase

is

described

in

Section

In

the

next

phase

we

train

a

HHMM

or

S-HSMM

over

the

alphabet

space

from

the

training

data

and

then

use

it

in

conjunction

with

the

Viterbi

to

perform

segmentation

and

annotation

The

architecture

of

the

framework

is

depicted

in

F

E

A

T

U

R

E

E

X

T

R

A

C

T

I

O

N

SHOT

DETECTION

AND

CLASSIFICATION

Direct

Narration

Assisted

Narration

Voice-Over

Expressive

Linkage

Functional

Linkage

M-phase

Coxian

M-phase

Coxian

M-phase

Coxian

M-phase

Coxian

M-phase

Coxian

END

Intro

main

body

Video

Audio

Signals

Figure

The

architecture

for

topic

detection

framework

The

two-level

HHMM

and

the

S-HSMM

-LRB-

whose

topology

is

shown

on

the

top

of

-RRB-

are

special

cases

of

the

hierarchical

model

with

two

layers

For

the

S-HSMM

-LRB-

HHMM

-RRB-

the

top

layer

is

a

Markov

sequence

of

switching

variables

while

the

bottom

layer

is

a

sequence

of

concate-nated

HSMMs

-LRB-

HMMs

-RRB-

whose

parameters

are

determined

by

the

switching

variables

at

the

top

Thus

the

dynamics

and

duration

parameters

of

the

HSMM

-LRB-

HMM

-RRB-

at

the

bottom

layer

are

not

time

invariant

but

are

switched

from

time

to

time

similar

to

the

way

the

linear

Gaussian

dynamics

are

switched

in

a

switching

Kalman

filter

When

mapping

to

the

topic

modeling

problem

the

bottom

layer

is

used

to

capture

atomic

semantics

such

as

voice-over

expressive

linkage

or

assisted

narration

Combinations

of

these

atomic

semantics

then

form

higher-level

semantics

each

of

which

is

represented

by

a

hidden

state

at

the

top

layer

in

our

model

The

Hierarchical

HMM

With

the

assumed

knowledge

of

the

flat

HMM

-LRB-

e.g.

see

-LSB-

-RSB-

-RRB-

we

shall

now

briefly

describe

the

HHMMs

A

hierarchical

HMM

is

formally

defined

by

a

three-turple

a

topo-logical

structure

parameterized

by

and

an

emission

alphabet

space

The

topology

specifies

the

model

depth

D

the

state

space

S

d

available

at

each

level

d

and

the

parent-children

relationship

between

two

consecutive

levels

For

example

the

two-level

topology

shown

on

the

top

of

specifies

the

children

set

at

the

bottom

level

for

state

is

-LCB-

-RCB-

and

for

state

is

-LCB-

-RCB-

Here

state

at

the

bottom

level

has

been

shared

by

both

state

and

at

the

top

level

Given

the

parameter

of

the

HHMM

is

specified

in

the

following

way

For

d

D

p

S

d

and

i

j

S

d

are

the

children

of

p

d

p

i

is

the

initial

probability

of

i

given

p

A

d

p

i

j

is

the

transition

probability

from

i

to

j

given

the

parent

p

and

A

d

p

i

end

is

the

probability

that

state

i

going

to

end-state

-LRB-

i.e.

returns

control

to

its

parent

-RRB-

given

the

parent

is

p.

Finally

for

each

state

i

at

the

lowest

level

D

and

an

alphabet

v

B

v

i

is

the

emission

probability

of

observing

v

given

the

current

state

at

the

lowest

level

is

i

The

whole

parameter

set

is

written

compactly

as

-LCB-

A

A

end

B

-RCB-

where

-LSB-

D

pSd

n

d

p

M

o

B

S

d

A

-LSB-

D

pSd

d

p

M

M

o

A

end

-LSB-

D

pSd

d

p

end

M

o

where

in

each

each

M

is

implicitly

meant

the

number

of

children

of

p

and

is

the

cardinality

operator

Stochastic

constraints

require

P

i

d

p

i

P

v

B

v

i

and

P

j

A

d

p

i

j

A

d

p

i

end

An

intuitive

way

to

view

the

set

is

to

consider

the

subset

-LCB-

d

p

A

d

p

A

d

p

end

-RCB-

as

the

parameter

of

the

p-initiated

Markov

chain

at

level

d

This

chain

is

terminated

when

one

of

the

children

i

of

p

reaches

the

end-state

with

the

probability

of

A

d

p

i

end

For

inference

and

learning

the

HHMM

is

represented

as

a

dynamic

Bayesian

network

-LRB-

DBN

-RRB-

and

can

be

learned

by

the

Asymmetric

Inside-Outside

algorithm

in

-LSB-

-RSB-

or

by

the

forward/backward

passes

in

-LSB-

-RSB-

shows

on

its

left

the

DBN

representation

of

the

HHMM

with

two

levels

i.e.

D

We

refer

readers

to

-LSB-

-RSB-

for

further

information

on

the

HHMMs

The

Switching-Hidden

Semi

Markov

Model

To

provide

an

intuitive

view

to

the

S-HSMM

starting

from

the

description

of

the

HHMMs

from

the

previous

section

let

us

consider

the

case

of

a

two-layer

HHMM

-LRB-

D

-RRB-

defined

as

follows

The

state

space

is

divided

into

the

set

of

states

at

the

top

level

Q

S

-LCB-

Q

-RCB-

and

states

at

the

bottom

level

Q

S

-LCB-

Q

-RCB-

This

model

is

parameterized

by

-LCB-

A

A

A

end

B

-RCB-

At

the

top

level

p

and

A

pq

are

respectively

the

initial

probability

and

the

transition

matrix

of

a

Markov

chain

defined

over

the

state

space

Q

For

each

p

Q

ch

-LRB-

p

-RRB-

Q

is

used

to

denote

the

set

of

children

of

p

As

in

the

case

of

the

extended

HHMM

in

-LSB-

-RSB-

it

is

possible

that

different

parent

states

may

share

common

children

i.e.

ch

-LRB-

p

-RRB-

ch

-LRB-

q

-RRB-

for

p

q

Q

A

transition

to

p

at

the

top

level

Markov

chain

will

initiate

a

sub-Markov

chain

at

the

lower

level

over

the

state

space

ch

-LRB-

p

-RRB-

parameterized

by

-LCB-

p

A

p

A

p

end

-RCB-

where

q

i

and

A

p

ij

are

the

initial

and

transition

probabilities

as

in

the

normal

HMM

setting

and

A

p

i

end

is

the

probability

that

this

chain

will

terminate

after

a

transition

to

i

At

each

time

point

t

a

discrete

symbol

y

t

is

generated

with

a

probability

of

B

v

i

where

i

is

the

current

state

at

the

bottom

level

In

the

description

of

this

two-level

HHMM

the

duration

d

for

which

a

bottom

state

i

remains

the

same

clearly

has

a

geometric

distribution

parameterized

by

its

non-self-transition

probability

-LRB-

A

p

ii

-RRB-

i.e.

d

Geom

-LRB-

A

p

ii

-RRB-

In

many

cases

the

geometric

distributions

are

often

too

restricted

to

model

realistic

data

The

Switching

Hidden

Semi-Markov

Models

-LRB-

S-HSMMs

-RRB-

proposed

in

-LSB-

-RSB-

overcomes

this

restriction

and

allows

the

duration

d

of

state

i

at

the

bottom

level

to

follow

a

more

general

discrete

distribution

d

D

p

i

d

More

precisely

the

p-initiated

chain

at

the

bottom

level

is

now

a

semi-Markov

sequence

parameterized

by

-LCB-

p

i

A

p

ij

D

p

i

d

-RCB-

as

opposed

to

the

normal

Markov

chain

in

the

HHMM

case

The

authors

in

-LSB-

-RSB-

consider

two

families

of

distributions

for

modeling

the

duration

the

multinomial

and

the

Coxian

However

for

the

multinomial

case

the

complexity

of

the

learning

algorithm

is

proportional

to

the

maximum

duration

length

thus

making

it

unsuitable

for

the

problem

of

modeling

video

data

which

is

usually

very

long

in

nature

Apart

from

the

disadvantage

of

assuming

a

maximum

duration

our

empirical

testing

on

the

multinomial

case

with

the

maximum

length

of

has

also

shown

that

it

is

about

times

slower

than

its

Coxian

counterpart

reported

in

this

paper

thus

making

it

impractical

in

our

settings

We

will

therefore

omit

the

multinomial

case

and

will

consider

exclu-sively

the

Coxian

parameterization

in

this

paper

A

discrete

M

phase

Coxian

distribution

Cox

-LRB-

-RRB-

parameterized

by

-LCB-

M

-RCB-

-LRB-

P

i

i

-RRB-

and

-LCB-

M

-RCB-

is

defined

as

a

mixture

of

P

M

i

i

S

i

where

S

i

-LRB-

X

i

X

M

-RRB-

in

which

X

i

are

independent

random

variables

having

geometric

distributions

X

i

Geom

-LRB-

i

-RRB-

This

distribution

is

a

member

of

the

phase-type

distribution

family

and

has

the

following

very

appealing

interpretation

Let

us

construct

a

Markov

chain

with

M

states

numbered

sequentially

with

the

self

transition

parameter

A

ii

i

as

shown

in

The

first

M

states

absorbing

state

M

M

M

M

Figure

The

phase

diagram

of

an

M

phase

Coxian

resent

M

phases

while

the

last

is

the

absorbing

state

which

acts

like

an

end

state

The

duration

of

each

individual

state

-LRB-

phase

-RRB-

i

is

X

i

Geom

-LRB-

i

-RRB-

If

we

start

from

state

i

the

duration

of

Markov

chain

before

the

end

state

reached

is

S

i

X

i

X

M

Thus

Cox

-LRB-

-RRB-

is

indeed

the

distribution

of

the

duration

of

this

constructed

Markov

chain

with

as

the

initial

state

-LRB-

phase

-RRB-

distribution

The

discrete

Coxian

is

much

more

flexible

than

the

geometric

distribution

its

probability

mass

function

is

no

longer

monotonically

decreasing

and

it

can

have

more

than

one

mode

Using

the

Coxian

distribution

the

duration

for

the

states

at

the

bottom

level

in

the

S-HSMM

is

modeled

as

follows

For

each

p-initiated

semi-Markov

sequence

the

duration

of

a

child

state

i

is

distributed

according

to

D

p

i

d

Cox

-LRB-

d

p

i

p

i

-RRB-

The

parameter

p

i

and

p

i

are

M

dimensional

vectors

where

M

is

a

fixed

number

representing

the

number

of

phases

in

the

discrete

Coxian

It

is

easy

to

verify

that

for

M

the

model

reduces

identically

to

a

two-layer

HHMM

Inference

and

Learning

in

the

S-HSMM

For

inference

and

learning

the

S-HSMM

is

represented

as

a

dynamic

Bayesian

network

as

shown

in

and

then

forward/backward

passes

are

applied

to

compute

the

filtering

and

smoothing

distributions

required

for

EM

learning

t

t

z

t

z

t

m

t

m

t

y

t

y

t

x

t

x

t

e

t

e

t

z

t

z

t

y

t

y

t

x

t

x

t

t

t

e

t

e

t

Figure

Two-slice

DBN

representation

of

a

two-level

HHMM

-LRB-

left

-RRB-

and

the

-LRB-

Coxian

-RRB-

S-HSMM

-LRB-

right

-RRB-

At

each

time-slice

t

an

amalgamated

hidden

state

S

t

-LCB-

z

t

t

x

t

e

t

m

t

-RCB-

together

with

the

observation

y

t

are

maintained

The

top

level

state

is

updated

via

z

t

and

t

is

a

boolean-valued

variable

set

to

when

the

z

t

initiated

semi-Markov

sequence

ends

at

t

At

the

bottom

level

x

t

is

the

current

child

state

in

the

z

t

initiated

chain

m

t

represents

the

current

phase

of

x

t

and

e

t

is

a

boolean-valued

variable

set

to

when

x

t

reaches

the

end

of

its

duration

The

forward

and

backward

procedures

in

the

general

DBN

are

then

used

to

compute

the

filtering

distribution

Pr

-LRB-

S

t

y

t

-RRB-

and

two

smoothing

distributions

Pr

-LRB-

S

t

y

T

-RRB-

and

Pr

-LRB-

S

t

S

t

y

T

-RRB-

With

these

smoothing

distributions

it

is

sufficient

to

derive

all

expected

sufficient

statistics

required

during

EM

learning

The

overall

complexity

for

the

forward

pass

-LRB-

and

also

for

the

EM

-RRB-

is

O

-LRB-

Q

Q

M

T

-RRB-

Further

information

can

be

found

in

-LSB-

-RSB-

Viterbi

decoding

for

segmentation

To

compute

the

best

sequence

state

that

is

to

find

S

T

argmax

S

T

Pr

-LRB-

S

T

y

T

-RRB-

Viterbi

decoding

algorithms

for

the

HHMM

and

S-HSMM

are

developed

These

algorithms

are

similar

to

the

one

used

in

the

standard

HMM

outlined

in

-LSB-

-RSB-

except

we

replace

the

normal

state

in

the

HMM

setting

by

our

amalgamated

state

S

t

which

-LCB-

z

t

x

t

t

m

t

e

t

-RCB-

for

the

S-HSMM

and

-LCB-

z

t

x

t

t

e

t

-RCB-

for

the

HHMM

-LRB-

cf.

-RRB-

In

this

section

we

detail

the

first

phase

in

the

detection

framework

This

includes

the

formulation

of

an

alphabet

set

for

shot

labeling

low-level

feature

extraction

and

shot

classification

Shot

labels

set

Existing

work

on

the

educational

videos

analysis

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

has

studied

the

nature

of

this

genre

carefully

As

noted

in

-LSB-

-RSB-

the

axiomatic

distinction

of

the

educational

genre

is

in

its

purpose

of

teaching

and

training

and

as

such

a

well-crafted

segment

that

moves

viewers

to

actions

or

retains

a

long-lasting

message

requires

elaborative

directing

skills

Based

on

a

narrative

analysis

used

in

the

educational

domain

and

observed

rules

and

conventions

in

the

production

of

this

media

the

authors

in

-LSB-

-RSB-

propose

a

hierarchy

of

narrative

structures

at

the

shot

level

as

shown

in

In

this

paper

we

select

the

five

most

meaningful

structures

from

this

hierarchy

for

experimentation

This

set

includes

direct-narration

-LRB-

DN

-RRB-

assisted-narration

-LRB-

AN

-RRB-

voice-over

-LRB-

VO

-RRB-

expressive-linkage

-LRB-

EL

-RRB-

and

functional-linkage

-LRB-

FL

-RRB-

We

shall

now

briefly

describe

these

narratives

Direct-narration

-LRB-

DN

-RRB-

and

assisted-narration

-LRB-

AN

-RRB-

are

re-ferred

to

jointly

as

on-screen

narration

which

refer

to

the

segments

with

the

appearance

of

the

narrator

The

purpose

of

these

sections

is

to

speak

to

the

viewers

with

the

voice

of

authority

and

is

commonly

used

to

demarcate

a

new

topic

or

subtopic

to

clarify

a

concept

or

to

lead

the

viewers

through

a

procedure

with

examples

DN

is

a

more

strict

form

of

on-screen

narration

It

involves

eye-to-eye

contact

where

the

narrator

speaks

to

the

viewers

directly

An

analogy

from

news

video

is

the

anchor-shot

AN

refers

to

parts

of

the

video

when

a

narrator

appears

in

a

more

diverse

style

and

the

attention

of

the

viewers

is

not

necessarily

focused

on

him

or

her

Here

the

purpose

is

not

only

to

talk

to

the

viewers

but

also

to

emphasize

a

message

by

means

of

text

captions

and/or

to

convey

an

experience

via

background

scenes

A

similar

structure

from

news

for

AN

is

the

reporting

shot

Assisted

narration

can

be

used

both

in

the

introduction

of

a

topic

or

in

the

main

body

and

thus

this

structure

should

be

shared

by

both

higher

semantics

introduction

and

main

body

As

we

see

later

this

knowledge

is

explicitly

modeled

and

incorporated

in

the

design

of

the

topology

for

the

S-HSMM

An

important

feature

is

that

although

the

semantics

of

AN

is

shared

the

typical

durations

are

different

when

it

is

used

in

the

introduction

or

the

main

body

respectively

An

AN

section

used

to

demarcate

a

new

topic

usually

contains

only

one

and

sometimes

two

shots

while

an

AN

section

used

in

the

main

body

is

typically

long

spanning

a

number

of

shots

Conditioning

on

the

parent

-LRB-

i.e.

introduction

or

main

body

-RRB-

the

typical

duration

distribution

of

the

AN

section

is

learned

automatically

for

each

case

by

our

model

The

voice-over

-LRB-

VO

-RRB-

structure

is

identified

as

sections

where

the

audiotrack

is

dominated

by

the

voice

of

the

narrator

but

without

his

or

her

appearance

The

purpose

of

these

segments

is

to

communicate

with

the

viewers

via

the

narrator

's

voice

Additional

pictorial

illustration

is

usually

further

shown

in

the

visual

channel

Expressive

linkage

-LRB-

EL

-RRB-

and

Functional

linkage

-LRB-

FL

-RRB-

belong

to

the

same

broader

linkage

group

in

the

hierarchy

in

The

purpose

of

the

linkage

structure

is

to

maintain

the

continuity

of

a

story

line

but

there

is

neither

on-screen

nor

voice-over

narration

involved

Functional

linkage

contains

transition

shots

encountered

in

switching

from

one

subject

to

the

next

Usually

large

superimposed

text

captions

are

used

and

the

voice

narration

is

completely

stopped

We

note

that

the

two

closest

video

genre

to

educational

videos

is

news

and

documentaries

In

the

description

of

what

follows

on

educational

genre

we

can

spot

several

similarities

across

these

genre

In

terms

of

parameterization

it

is

a

form

of

parameter

tying

Linkage

Narration

On-screen

Narration

S

u

p

p

o

r

t

i

v

e

N

a

r

r

a

t

i

o

n

N

a

r

r

a

t

i

o

n

V

o

i

c

e

O

v

e

r

vo

educational

videos

on

lk

f

u

lk

e

x

lk

Ex

pre

ssi

ve

Lin

kag

e

Fu

nct

ion

al

Lin

kag

e

sn

Supportive

Narration

Di

r

ect

i

on

r

rat

i

on

an

w

t

dn

a

n

w

s

vo

w.

Texts/Scenes

vo

w

ith

Sc

enes

v

o

t

s

vo

w

s

vo

w

t

Ass

rrw

S

cen

es

As

sN

arr

w.

Te

xts

vo

with

Texts

Figure

The

hierarchy

of

narrative

structures

in

educational

videos

proposed

in

-LSB-

-RSB-

with

possibly

music

played

in

the

background

Expressive

linkage

on

the

other

hand

is

used

to

create

mood

for

the

subject

being

presented

For

example

in

the

video

presenting

the

fire

safety

topic

there

is

a

segment

in

which

the

narration

is

completely

stopped

and

then

a

sequence

of

pictures

of

the

house

on

fire

is

shown

These

scenes

obviously

do

not

give

any

direct

instruction

rather

they

create

a

sense

of

mood

that

helps

the

video

to

be

more

appealing

and

interesting

Feature

extraction

and

shot

classification

The

feature

set

and

method

for

shot

classification

described

in

-LSB-

-RSB-

is

employed

in

this

paper

The

feature

set

is

extracted

from

both

visual

and

audio

streams

at

the

shot-based

level

From

the

image

sequence

we

choose

to

detect

the

frontal

faces

to

reflect

the

appearance

of

the

narrator

using

the

CMU

face

detection

algoritm

-LSB-

-RSB-

and

captioned

texts

as

one

of

the

common

means

of

conveying

information

in

educational

videos

using

the

algorithm

described

in

-LSB-

-RSB-

In

order

to

classify

a

shot

into

direct-narration

voice-over

linkage

etc.

further

information

is

sought

from

the

audio

stream

Audio

features

are

computed

as

the

percentage

of

the

following

audio

classes

within

a

shot

vocal

speech

music

silence

and

non-literal

sound

A

shot

is

then

classified

into

one

of

the

elements

of

-LCB-

DN

AN

V

O

EL

F

L

-RCB-

using

the

classification

framework

reported

in

-LSB-

-RSB-

Since

we

claim

no

contribution

at

this

stage

we

shall

refer

readers

to

-LSB-

-RSB-

for

full

details

on

this

classification

scheme

Our

dataset

D

consists

of

educational

and

training

videos

containing

different

types

of

subjects

and

presentational

styles

and

thus

this

constitutes

a

relatively

noisy

set

of

data

We

manually

provide

groundtruth

for

these

videos

with

topic

transitions

In

some

cases

the

groundtruth

for

topic

transitions

comes

directly

from

the

hardcopy

guidelines

supplied

by

the

producer

At

the

pre-processing

stage

Webflix

-LSB-

-RSB-

is

used

to

perform

shot

transition

detection

and

all

detection

errors

are

corrected

manually

Since

our

contribution

from

this

paper

is

at

the

semantic

level

the

latter

step

is

to

ensure

an

error

at

the

shot

detection

does

not

influence

the

performance

of

the

system

at

higher

levels

Since

educational

videos

mainly

contain

cut

and

dissolve

transitions

the

shot

detection

accuracy

is

found

to

be

very

high

with

rare

cases

being

erroneous

Given

shot

indices

each

video

is

processed

as

described

in

Section

and

then

each

shot

S

is

labeled

as

one

of

the

elements

of

-LCB-

DN

AN

V

O

EL

F

L

-RCB-

Model

topology

and

parameterization

We

will

use

four

models

in

this

experiments

the

flat

HMM

and

HSMM

-LRB-

as

the

baseline

cases

-RRB-

the

HHMM

and

the

S-HSMM

For

the

flat

HMM

and

HSMM

we

range

the

number

of

states

from

to

with

the

observation

space

where

is

intended

to

be

the

minimum

number

of

states

required

-LRB-

like

intro

and

main

body

-RRB-

and

is

the

number

of

alphabets

-LRB-

i.e.

in

the

relaxed

way

that

the

number

of

states

equates

to

the

number

of

alphabets

-RRB-

The

semi-Markov

version

HSMM

is

further

parameterized

by

Coxian

distributions

as

the

duration

distributions

of

the

states

The

choice

of

M

phases

is

hinted

by

the

results

reported

in

-LSB-

-RSB-

where

M

has

resulted

in

best

performances

For

the

HHMM

and

the

S-HSMM

the

topology

shown

in

the

top

of

is

used

to

construct

the

S-HSMM

in

this

experiment

This

topology

specifies

Q

states

at

the

top

level

where

state

and

correspond

to

the

introduction

and

the

main

body

of

the

topic

respectively

The

Markov

chain

at

this

level

is

similar

to

the

flat

HMM

used

in

-LSB-

-RSB-

for

news

story

segmentation

reviewed

in

Section

We

incorporate

the

assumed

prior

knowledge

that

a

topic

usually

starts

with

either

direct-narration

assisted-narration

or

functional

linkage

thus

state

has

-LCB-

-RCB-

as

its

children

set

Similarly

the

main

body

can

contain

assisted-narration

voice-over

or

expressive

linkage

hence

its

children

set

is

-LCB-

-RCB-

Here

state

-LRB-

assisted

narration

-RRB-

has

been

shared

by

both

parent

state

-LRB-

intro

-RRB-

and

-LRB-

main

body

-RRB-

The

bottom

level

has

states

corresponding

to

shot

labels

To

map

the

labels

to

the

bottom

states

we

construct

a

diagonal-like

B

observation

matrix

and

fix

it

i.e.

we

do

not

learn

B

The

diagonal

entries

of

B

are

set

to

to

relax

the

uncertainty

during

the

classification

stage

The

duration

models

in

the

S-HSMM

are

used

with

M

phases

Coxian

Detection

Results

Given

the

dataset

D

our

evaluation

employs

a

leave-one-out

strategy

to

ensure

an

objective

cross-validation

We

sequentially

pick

out

a

video

V

and

use

the

remainder

set

-LCB-

D

V

-RCB-

to

train

the

model

and

then

use

V

for

testing

In

the

results

that

follow

this

method

is

used

for

all

cases

including

the

flat

HMM

the

flat

HSMM

hierarchical

HMM

and

the

S-HSMM

A

topic

transition

is

detected

when

the

introduction

state

at

the

top

level

is

reached

during

the

Viterbi

decoding

Examples

of

Viterbi

decoding

with

the

S-HSMM

and

HHMM

are

shown

in

To

measure

the

performance

in

addition

to

the

well-known

They

called

transition

and

internal

states

instead

of

introduction

and

main

body

recall

-LRB-

recall

-RRB-

and

precision

-LRB-

prec

-RRB-

metrics

we

include

the

F-score

-LRB-

f-score

-RRB-

metric

defined

as

f-score

recall

prec

recall

prec

recall

prec

While

the

recall

rate

measures

how

well

the

system

can

recover

the

true

topic

transitions

and

high

precision

ensures

that

it

does

not

over-segment

the

video

the

F-score

shows

the

overall

performance

of

the

system

In

the

ideal

case

when

recall

prec

clearly

f-score

i.e.

the

highest

performance

the

system

can

achieve

The

baseline

cases

flat

HMM

and

HSMM

Since

initialization

is

crucial

during

EM

learning

we

apply

multiple

random

restart

points

when

conducting

the

experiments

including

the

uniform

initialization

Although

several

restarts

were

used

the

flat

HMM

is

found

to

yield

extremely

poor

results

in

all

cases

Even

when

we

train

and

test

on

the

same

dataset

the

flat

HMM

still

produces

poor

detection

results

proving

to

be

unsuitable

in

our

topical

transition

detection

settings

The

flat

HSMM

produces

slightly

better

results

than

the

flat

HMM

but

still

in

all

ten

runs

the

performance

is

still

very

low

-LRB-

recall

and

prec

in

the

best

case

-RRB-

The

poor

performance

of

the

HMM

and

HSMM

is

of

no

surprise

since

their

forms

are

too

strict

to

model

a

rather

high

concept

the

topic

Furthermore

with

the

flat

structures

they

offer

no

mechanism

to

incorporate

prior

domain

knowledge

such

as

those

that

we

use

in

the

topology

of

the

S-HSMM

and

HHMM

This

clearly

shows

that

hierarchical

models

are

much

more

suitable

for

video

analysis

than

the

flat

ones

Given

the

poor

results

in

the

flat

structure

cases

we

will

omit

the

HMM

and

HSMM

in

the

discussion

of

what

follows

below

Detection

with

the

S-HSMM

and

HHMM

The

recall

rate

precision

and

F-score

for

representative

runs

are

reported

in

Table

in

which

the

best

performance

are

highlighted

in

bold

The

detection

results

for

each

individual

video

for

the

best

cases

are

shown

in

Table

With

different

random

restarting

points

including

the

uniform

initialization

the

performance

of

the

HHMM

ranges

from

poor

to

very

good

-LRB-

for

recall

and

for

precision

-RRB-

whereas

the

S-HSMM

consistently

yields

good

results

-LRB-

for

recall

and

for

precision

-RRB-

Since

during

training

there

is

nothing

exposed

to

the

testing

examples

we

also

report

-LRB-

in

the

second

part

of

Table

-RRB-

the

performances

of

the

HHMM

and

S-HSMM

in

a

likelihood-based

best

model

selection

scheme

This

scheme

works

as

follows

As

in

the

leave-one-out

strategy

let

V

be

a

video

selected

from

D

and

N

is

the

number

of

times

we

train

the

model

using

the

dataset

-LCB-

D

V

-RCB-

-LRB-

i.e.

without

V

-RRB-

Let

i

-LRB-

V

-RRB-

and

L

i

-LRB-

V

-RRB-

-LRB-

i

N

-RRB-

respectively

be

the

learned

model

and

the

likelihood

-LRB-

at

convergence

-RRB-

obtained

for

i-th

run

We

then

use

the

model

i

to

test

on

the

unseen

video

V

where

i

argmax

i

N

L

i

-LRB-

V

-RRB-

Simply

speaking

we

sequentially

throw

away

a

video

V

then

select

the

best

model

-LRB-

i.e.

highest

likelihood

-RRB-

among

all

runs

to

test

on

V

For

the

HHMM

the

result

stays

the

same

as

when

we

choose

the

best

performance

based

on

the

F-score

For

the

S-HSMM

the

recall

stays

the

same

while

the

precision

slightly

decreases

from

to

Nevertheless

the

S-HSMM

is

still

superior

to

the

HHMM

recall

-LRB-

-RRB-

prec

-LRB-

-RRB-

f-score

results

for

best

performance

selection

Uniform

Rand

HHMM

Rand

Rand

Rand

Rand

Uniform

Rand

S-HSMM

Rand

Rand

Rand

Rand

results

for

best

model

selection

HHMM

S-HSMM

Table

Detection

Performances

for

the

S-HSMM

and

the

HHMM

Best

performance

for

each

case

is

highlighted

in

bold

-LRB-

we

note

that

best

performances

are

attained

in

multiple

cases

and

we

select

one

of

them

to

highlight

-RRB-

Table

and

show

that

modeling

with

the

S-HSMM

results

in

better

performances

than

the

HHMM

in

both

recall

and

precision

rates

And

as

a

result

the

F-score

improves

from

to

While

the

recall

rate

improves

only

slightly

the

improvement

in

the

precision

indicates

that

the

HHMM

tends

to

over-segment

the

video

more

frequently

than

the

S-HSMM

This

has

confirmed

our

belief

that

duration

information

is

an

important

factor

in

our

topic

transition

detection

settings

The

semi-Markov

modeling

has

effectively

overcome

the

limitation

of

the

strict

Markov

assumption

of

-LCB-

future

past

present

-RCB-

in

the

flat

HMM

allowing

longer

temporal

dependency

to

be

captured

via

the

duration

of

the

state

Nevertheless

given

a

somewhat

more

contained

set

of

data

used

in

this

experiment

the

results

from

both

the

S-HSMM

and

HHMM

are

better

than

the

previous

detection

results

of

news

story

reported

in

-LSB-

-RSB-

-LRB-

which

came

first

in

testbed

-RRB-

and

the

heuristics

and

Bayesian

approaches

on

topic

detection

in

-LSB-

-RSB-

These

results

do

not

only

imply

the

advantages

of

the

S-HSMM

over

the

HHMM

but

also

show

the

contribution

of

the

HHMM

in

its

own

right

In

this

paper

we

explore

the

difficult

problem

of

detecting

topic

transitions

through

the

use

of

two

probabilistic

models

the

HHMM

and

the

S-HSMM

Both

allow

the

modeling

of

hierarchy

and

the

sharing

of

substructures

within

the

hierarchy

whilst

the

S-HSMM

additionally

allows

the

explicit

modeling

of

durative

properties

Coupled

with

the

use

of

the

Coxian

model

we

show

how

this

unified

framework

performs

better

than

the

baseline

cases

-LRB-

the

flat

HMM

and

HSMM

-RRB-

and

previous

results

reported

In

particular

the

use

of

the

S-HSMM

demonstrates

that

the

modeling

of

duration

is

a

i.e.

the

future

is

conditionally

independent

of

the

past

given

the

present

Video

TP

FP

Miss

GT

EESafety

SSFall

ElectS

TrainHaz

EyeS

FootS

HKeeping

Maintn

HandS

SBurning

HeadProt

WeldingS

Sum

Table

Detection

results

for

each

video

in

the

best

performance

cases

of

the

S-HSMM

and

the

HHMM

-LRB-

TP

True

Positive

FP

False

Positive

GT

Ground-Truth

-RRB-

powerful

tool

in

the

extraction

of

higher

level

semantics

The

results

demonstrate

the

promise

of

the

approach

and

although

the

results

are

demonstrated

with

the

educational

and

training

film

genre

the

method

can

easily

be

applied

to

other

genres

We

believe

that

the

promise

of

the

approach

lies

in

its

unified

probabilistic

handling

of

durative

properties

and

shared

hierarchical

structure

allowing

it

to

handle

long

video

sequences

with

inherent

variability

and

complicated

semantics

Hung

Bui

is

supported

by

the

Defense

Advanced

Research

Projects

Agency

-LRB-

DARPA

-RRB-

through

the

Department

of

the

Interior

NBC

Acquisition

Services

Division

under

Contract

No

-LSB-

-RSB-

B.

Adams

C.

Dorai

and

S.

Venkatesh

Automated

film

rhythm

extraction

for

scene

analysis

In

IEEE

International

Conference

on

Multimedia

and

Expo

pages

Tokyo

Japan

August

-LSB-

-RSB-

P.

Aigrain

P.

Jolly

and

V.

Longueville

Medium

knowledge-based

macro-segmentation

of

video

into

sequences

In

M.

Maybury

editor

Intelligent

Multimedia

Information

Retrieval

pages

AAAI

Press/MIT

Press

-LSB-

-RSB-

H.

H.

Bui

D.

Q.

Phung

and

S.

Venkatesh

Hierarchical

hidden

markov

models

with

general

state

hierarchy

In

D.

L.

McGuinness

and

G.

Ferguson

editors

Proceedings

of

the

Nineteenth

National

Conference

on

Artificial

Intelligence

pages

San

Jose

California

USA

AAAI

Press

The

MIT

Press

-LSB-

-RSB-

L.

Chaisorn

T.-S

Chua

C.-H

Lee

and

Q.

Tian

A

hierarchical

approach

to

story

segmentation

of

large

broadcast

news

video

corpus

In

IEEE

International

Conference

on

Multimedia

and

Expo

Taipei

Taiwan

June

-LSB-

-RSB-

T.

V.

Duong

H.

H.

Bui

D.

Q.

Phung

and

S.

Venkatesh

Activity

recognition

and

abnormality

detection

with

the

Switching

Hidden

Semi-Markov

Model

In

IEEE

Int

Conf

on

Computer

Vision

and

Pattern

Recognition

volume

pages

San

Diego

June

IEEE

Computer

Society

-LSB-

-RSB-

S.

Fine

Y.

Singer

and

N.

Tishby

The

hierarchical

hidden

markov

model

Analysis

and

applications

Machine

Learning

-LRB-

-RRB-

-LSB-

-RSB-

A.

Hanjalic

Shot-boundary

detection

Unraveled

and

resolved

IEEE

Transaction

in

Circuits

and

Systems

for

Video

Technology

-LRB-

-RRB-

-LSB-

-RSB-

A.

Hanjalic

R.

L.

Lagendijk

and

J.

Biemond

Automated

high-level

movie

segmentation

for

advanced

video

retrieval

systems

IEEE

Transactions

in

Circuits

and

Systems

for

Video

Technology

-LRB-

-RRB-

-LSB-

-RSB-

I.

Ide

K.

Yamamoto

and

H.

Tanaka

Automatic

video

indexing

based

on

shot

classification

In

First

International

Conference

on

Advanced

Multimedia

Content

Processing

pages

Osaka

Japan

November

-LSB-

-RSB-

U.

Iurgel

R.

Meermeier

S.

Eickeler

and

G.

Rigoll

New

approaches

to

audio-visual

segmentation

of

TV

news

for

automatic

topic

retrieval

In

IEEE

Int

Conf

on

Acoustics

Speech

and

Signal

Processing

volume

pages

Salt

Lake

City

Utah

-LSB-

-RSB-

E.

Kijak

L.

Oisel

and

P.

Gros

Hierarchical

structure

analysis

of

sport

videos

using

HMMs

In

Int

Conf

on

Image

Processing

volume

pages

II

vol

-LSB-

-RSB-

S.

E.

Levinson

Continuously

variable

duration

hidden

markov

models

for

automatic

speech

recognition

Computer

Speech

and

Language

-LRB-

-RRB-

March

-LSB-

-RSB-

T.

Lin

and

H.

J.

Zhang

Automatic

video

scene

extraction

by

shot

grouping

Pattern

Recognition

-LSB-

-RSB-

Z.

Liu

and

Q.

Huang

Detecting

news

reporting

using

audio/visual

information

In

International

Conference

on

Image

Processing

pages

Kobe

Japan

October

-LSB-

-RSB-

Mediaware-Company

Mediaware

solution

webflix

professional

-LSB-

-RSB-

C.

D.

Mitchell

and

L.

H.

Jamieson

Modeling

duration

in

a

hidden

markov

model

with

the

exponential

family

In

Proc

of

IEEE

Int

Conf

on

Acoustics

Speech

and

Signal

Processing

pages

II

II

Minneapolis

Minnesota

April

-LSB-

-RSB-

K.

Murphy

and

M.

Paskin

Linear-time

inference

in

hierarchical

HMMs

In

T.

G.

Dietterich

S.

Becker

and

Z.

Ghahramani

editors

Advances

in

Neural

Information

Processing

Systems

Cambridge

MA

MIT

Press

-LSB-

-RSB-

M.

R.

Naphade

and

T.

S.

Huang

Discovering

recurrent

events

in

video

using

unsupervised

methods

In

Int

Conf

om

Image

Processing

volume

pages

Rochester

NY

USA

-LSB-

-RSB-

D.

Q.

Phung

Probabilistic

and

Film

Grammar

Based

Methods

for

Video

Content

Analysis

PhD

thesis

Curtin

University

of

Technology

Australia

-LSB-

-RSB-

D.

Q.

Phung

H.

H.

Bui

and

S.

Venkatesh

Content

structure

discovery

in

educational

videos

with

shared

Detected

Topic

Transitions

z

t

x

t

m

t

t

e

t

main

body

intro

shot

number

z

t

x

t

t

e

t

S

H

S

M

M

H

H

M

M

Detected

Topic

Transitions

Ground-Truth

Figure

Example

of

Viterbi

decoding

for

the

S-HSMM

and

the

HHMM

for

the

first

shots

of

video

EESafety

These

results

should

be

read

together

with

to

see

the

semantics

of

the

DBN

structure

structures

in

the

hierarchical

HMMs

In

Joint

Int

Workshop

on

Syntactic

and

Structural

Pattern

Recognition

pages

Lisbon

Portugal

August

-LSB-

-RSB-

D.

Q.

Phung

and

S.

Venkatesh

Structural

unit

identification

and

segmentation

of

topical

content

in

educational

videos

Technical

report

Department

of

Computing

Curtin

University

of

Technology

-LSB-

-RSB-

D.

Q.

Phung

S.

Venkatesh

and

H.

H.

Bui

Automatically

learning

structural

units

in

educational

videos

using

the

hierarchical

HMMs

In

International

Conference

on

Image

Processing

Singapore

-LSB-

-RSB-

D.

Q.

Phung

S.

Venkatesh

and

C.

Dorai

High

level

segmentation

of

instructional

videos

based

on

the

content

density

function

In

ACM

International

Conference

on

Multimedia

pages

Juan

Les

Pins

France

December

-LSB-

-RSB-

L.

R.

Rabiner

A

tutorial

on

hidden

markov

models

and

selected

applications

in

speech

recognition

In

Procs

IEEE

volume

pages

February

-LSB-

-RSB-

H.

A.

Rowley

S.

Baluja

and

T.

Kanade

Neutral

network-based

face

detection

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

January

-LSB-

-RSB-

K.

Shearer

C.

Dorai

and

S.

Venkatesh

Incorporating

domain

knowlege

with

video

and

voice

data

analysis

In

Workshop

on

Multimedia

Data

Minning

Boston

USA

August

-LSB-

-RSB-

J.-C

Shim

C.

Dorai

and

R.

Bolle

Automatic

text

extraction

from

video

for

content-based

annotation

and

retrieval

In

International

Conference

on

Pattern

Recognition

volume

pages

Brisbane

Australia

August

-LSB-

-RSB-

C.

G.

Snoek

and

M.

Worring

Multimodal

video

indexing

A

review

of

the

state-of-the-art

Multimedia

Tools

and

Applications

In

Press

-LSB-

-RSB-

H.

Sundaram

Segmentation

Structure

Detection

and

Summarization

of

Multimedia

Sequences

PhD

thesis

Columbia

University

-LSB-

-RSB-

H.

Sundaram

and

S.-F

Chang

Computable

scenes

and

structures

in

films

IEEE

Transactions

in

Multimedia

-LRB-

-RRB-

-LSB-

-RSB-

B.

T.

Truong

An

Investigation

into

Structural

and

Expressive

Elements

in

Film

PhD

thesis

Curtin

University

of

Technology

-LSB-

-RSB-

J.

Vendrig

and

M.

Worring

Systematic

evaluation

of

logical

story

unit

segmentation

IEEE

Transactions

on

Multimedia

-LRB-

-RRB-

-LSB-

-RSB-

C.

Wang

Y.

Wang

H.

Liu

and

Y.

He

Automatic

story

segmentation

of

news

video

based

on

audio-visual

features

and

text

information

In

Int

Conf

on

Machine

Learning

and

Cybernetics

volume

pages

-LSB-

-RSB-

J.

Wang

T.-S

Chua

and

L.

Chen

Cinematic-based

model

for

scene

boundary

detection

In

The

Eight

Conference

on

Multimedia

Modeling

Amsterdam

Netherland

November

-LSB-

-RSB-

L.

Xie

and

S.-F

Chang

Unsupervised

mining

of

statistical

temporal

structures

in

video

In

A.

Rosenfield

D.

Doreman

and

D.

Dementhons

editors

Video

Mining

Kluwer

Academic

Publishers

June

-LSB-

-RSB-

L.

Xie

S.-F

Chang

A.

Divakaran

and

H.

Sun

Learning

hierarhical

hidden

markov

models

for

unsupervised

structure

discovery

from

video

Technical

report

Columbia

University

-LSB-

-RSB-

X.

Zhu

L.

Wu

X.

Xue

X.

Lu

and

J.

Fan

Automatic

scene

detection

in

news

program

by

integrating

visual

feature

and

rules

In

IEEE

Pacific-Rim

Conference

on

Multimedia

pages

Beijing

China

extended hierarchical hidden markov model

unified and coherent probabilistic framework

typical durations of important semantics

coxian switching hidden semi-markov model

hierarchical markov (semi-markov) models

natural hierarchical organization of videos

semantic relationship of neighborhood scenes


Robustness Analysis of Cognitive Information Complexity Measure using Weyuker Properties Cognitive information complexity measure is based on cognitive informatics , which helps in comprehending the software characteristics . For any complexity measure to be robust , Weyuker properties must be satisfied to qualify as good and comprehensive one . In this paper , an attempt has also been made to evaluate cognitive information complexity measure in terms of nine Weyuker properties , through examples . It has been found that all the nine properties have been satisfied by cognitive information complexity measure and hence establishes cognitive information complexity measure based on information contained in the software as a robust and well-structured one . Keywords : Weighted information count , cognitive information complexity unit , cognitive weight , cognitive information complexity measure , basic control structures . Many well known software complexity measures have been proposed such as McCabe 's cyclomatic number -LSB- 8 -RSB- , Halstead programming effort -LSB- 5 -RSB- , Oviedo 's data flow complexity measures -LSB- 9 -RSB- , Basili 's measure -LSB- 3 -RSB- -LSB- 4 -RSB- , Wang 's cognitive complexity measure -LSB- 11 -RSB- and others -LSB- 7 -RSB- . All the reported complexity measures are supposed to cover the correctness , effectiveness and clarity of software and also to provide good estimate of these parameters . Out of the numerous proposed measures , selecting a particular complexity measure is again a problem , as every measure has its own advantages and disadvantages . There is an ongoing effort to find such a comprehensive complexity measure , which addresses most of the parameters of software . Weyuker -LSB- 14 -RSB- has suggested nine properties , which are used to determine the effectiveness of various software complexity measures . A good complexity measure should satisfy most of the Weyuker 's properties . A new complexity measure based on weighted information count of a software and cognitive weights has been developed by Kushwaha and Misra -LSB- 2 -RSB- . In this paper an effort has been made to estimate this cognitive information complexity measure as robust and comprehensive one by evaluating this against the nine Weyuker 's properties . Basic control structures -LSB- BCS -RSB- such as sequence , branch and iteration -LSB- 10 -RSB- -LSB- 13 -RSB- are the basic logic building blocks of any software and the cognitive weights -LRB- W c -RRB- of a software -LSB- 11 -RSB- is the extent of difficulty or relative time and effort for comprehending a given software modeled by a number of BCS 's . These cognitive weights for BCS 's measure the complexity of logical structures of the software . Either all the BCS 's are in a linear layout or some BCS 's are embedded in others . For the former case , we sum the weights of all the BCS 's and for the latter , cognitive weights of inner BCS 's are multiplied with the weight of external BCS 's . Since software represents computational information and is a mathematical entity , the amount of information contained in the software is a function of the identifiers that hold the information and the operators that perform the operations on the information i.e. Information = f -LRB- Identifiers , Operators -RRB- Identifiers are variable names , defined constants and other labels in a software . Therefore information contained in one line of code is the number of all operators and operands in that line of code . Thus in k th line of code the Information contained is : I k = -LRB- Identifiers + Operands -RRB- k = -LRB- ID k + OP k -RRB- IU Where ID = Total number of identifiers in the k th LOC of software , OP = Total number of operators in the k th LOC of software , IU is the Information Unit representing that at least any identifier or operator has one unit information in them . Total Information contained in a software -LRB- ICS -RRB- is sum of information contained in each line of code i.e. LOCS ICS = -LRB- I k -RRB- k = 1 Where I k = Information contained in k th line of code , LOCS = Total lines of code in the software . Thus , it is the information contained in the identifiers and the necessary operations carried out by the operators in achieving the desired goal of the software , which makes software difficult to understand . Once we have established that software can be comprehended as information defined in information units -LRB- IU 's -RRB- -LSB- 2 -RSB- , the weighted information count is defined as : The Weighted Information Count of a line of code -LRB- WICL -RRB- of a software is a function of identifiers , operands and LOC and is defined as : ACM SIGSOFT Software Engineering Notes Page 1 January 2006 Volume 31 Number 1 ACM SIGSOFT Software Engineering Notes Page 1 January 2006 Volume 31 Number 1 WICL k = ICS k / -LSB- LOCS k -RSB- Where WIC k = Weighted Information Count for the k th line , ICS k = Information contained in a software for the k th line . The Weighted Information Count of the Software -LRB- WICS -RRB- is defined as : LOCS WICS = WICL k k = 1 In order to be a complete and robust measure , the measure of complexity should also consider the internal control structure of the software . These basic control structures have also been considered as the Newton 's law in software engineering -LSB- 10 , 11 -RSB- . These are a set of fundamental and essential flow control mechanisms that are used for building the logical architectures of software . Using the above definitions , Cognitive Information Complexity Measure -LRB- CICM -RRB- is defined as the product of weighted information count of software -LRB- WICS -RRB- and the cognitive weight -LRB- W c -RRB- of the BCS in the software i.e. CICM = WICS * W c This complexity measure encompasses all the major parameters that have a bearing on the difficulty in comprehending software or the cognitive complexity of the software . It clearly establishes a relationship between difficulty in understanding software and its cognitive complexity . It introduces a method to measure the amount of information contained in the software thus enabling us to calculate the coding efficiency -LRB- E I -RRB- as ICS / LOCS -LSB- 2 -RSB- . Weyuker -LSB- 14 -RSB- proposed the nine properties to evaluate any software complexity measure . These properties also evaluate the weakness of a measure in a concrete way . With the help of these properties one can determine the most suitable measure among the different available complexity measures . In the following paragraphs , the cognitive information complexity measure has been evaluated against the nine Weyuker properties for establishing itself as a comprehensive measure . Property 1 : -LRB- P -RRB- -LRB- Q -RRB- -LRB- | P | | Q | -RRB- Where P and Q are program body . This property states that a measure should not rank all programs as equally complex . Now consider the following two examples given in Fig. 1 and Fig. 2 . For the program given in Fig. 1 in Appendix I , there are two control structures : a sequential and a iteration . Thus cognitive weight of these two BCS 's is 1 + 3 = 4 . Weighted information count for the above program is as under : WICS = 3/6 + 1/4 + 6/3 + 4/2 = 4.75 Hence Cognitive information complexity measure -LRB- CICM -RRB- is : CICM = WICS * W c = 4.75 * 4 = 19.0 For the program given in Fig. 2 in Appendix I there is only one sequential structure and hence the cognitive weight W c is 1 . WICS for the above program is 2.27 . Hence CICM for the above program is 2.27 * 1 = 2.27 . From the complexity measure of the above two programs , it can be seen that the CICM is different for the two programs and hence satisfies this property . Property 2 : Let c be a non-negative number . Then there are only finitely many programs of complexity c. Calculation of WICS depends on the number of identifiers and operators in a given program statement as well as on the number of statements remaining that very statement in a given program . Also all the programming languages consist of only finite number of BCS 's . Therefore CICM can not rank complexity of all programs as c. Hence CICM holds for this property . Property 3 : There are distinct programs P and Q such that | P | = | Q | . For the program given in Fig. 3 in Appendix I , the CICM for the program is 19 , which is same as that of program in Fig. 1 . Therefore this property holds for CICM . Property 4 : -LRB- P -RRB- -LRB- Q -RRB- -LRB- PQ & | P | | Q | -RRB- Referring to program illustrated in Fig. 1 , we have replaced the while loop by the formula `` sum = -LRB- b +1 -RRB- * b/2 '' and have illustrated the same in Fig. 2 . Both the programs are used to calculate the sum of first n integer . The CICM for both the programs is different , thus establishing this property for CICM . Property 5 : -LRB- P -RRB- -LRB- Q -RRB- -LRB- | P | | P ; Q | and | Q | | P ; Q | -RRB- . Consider the program body given in Fig. 4 in Appendix I : The program body for finding out the factorial of a number consists of one sequential and one branch BCS 's . Therefore W c = 3 . For the program body for finding out the prime number , there are one sequential , one iteration and two branch BCS 's . Therefore W c = 1 + 2 * 3 * 2 = 13 . For the main program body for finding out the prime and factorial of the number , there are one sequential , two call and one branch BCS 's . Therefore W c = 1 +5 +15 +2 = 23 . WICS for the program is 5.1 . Therefore the Cognitive Information Complexity Measure for the above program = 5.1 * 23 = 117.3 . Now consider the program given in Fig. 5 in Appendix I to check for prime . There is one sequential , one iteration and three branch BCS 's . Therefore W c = 1 + 2 * 3 * 2 + 2 = 15 . WICS = 1.85 . So CICM = 1.85 * 15 = 27.79 . For the program given in Fig. 6 in Appendix I , there is one sequential , one iteration and one branch BCS 's . Wc for this program is 7 and WICS is .5.11 . Hence CICM = WICS * W c = 5.11 * 7 = 35.77 . It is clear from the above example that if we take the two-program body , one for calculating the factorial and another for checking for ACM SIGSOFT Software Engineering Notes Page 2 January 2006 Volume 31 Number 1 prime whose CICM are 27.79 and 35.77 that are less than 117.3 . So property 5 also holds for CICM . Property 6 -LRB- a -RRB- : -LRB- P -RRB- -LRB- Q -RRB- -LRB- R -RRB- -LRB- | P | = | Q | -RRB- & -LRB- | P ; R | | Q ; R | -RRB- Let P be the program illustrated in Fig. 1 and Q is the program illustrated in Fig. 3 . The CICM of both the programs is 19 . Let R be the program illustrated in Fig. 6 . Appending R to P we have the program illustrated in Fig. 7 in Appendix I. Cognitive weight for the above program is 9 and WICS is 8.3 . Therefore CICM = 8.3 * 9 = 74.7 . Similarly appending R to Q we have W c = 9 and WICS = 8.925 . Therefore CICM = 8.925 * 9 = 80.325 and 74.7 80.325 . This proves that Property 6 -LRB- a -RRB- holds for CICM . Property 6 -LRB- b -RRB- : -LRB- P -RRB- -LRB- Q -RRB- -LRB- R -RRB- -LRB- | P | = | Q | -RRB- & -LRB- | R ;P | | R : Q | -RRB- To illustrate the above property let us arbitrarily append three program statements in the programs given in Fig. 1 , we have the program given in Fig. 8 in Appendix I . There is only one sequential and one iteration BCS . Hence cognitive weight is 1 + 3 = 4 . There is only one sequential and one iteration BCS . Hence cognitive weight is 1 + 3 = 4 and WICS = 5.58 . So CICM = 5.58 * 4 = 22.32 . Similarly appending the same three statements to program in Fig. 3 we again have cognitive weights = 4 and WICS = 5.29 . Therefore CICM = 21.16 22.32 . Hence this property also holds for CICM . Property 7 : There are program bodies P and Q such that Q is formed by permuting the order of the statement of P and -LRB- | P | | Q | -RRB- . Since WICS is dependent on the number of operators and operands in a given program statement and the number of statements remaining after this very program statement , hence permuting the order of statement in any program will change the value of WICS . Also cognitive weights of BCS 's depend on the sequence of the statement -LSB- 1 -RSB- . Hence CICM will be different for the two programs . Thus CICM holds for this property also . Property 8 : If P is renaming of Q , then | P | = | Q | . CICM is measured in numeric and naming or renaming of any program has no impact on CICM . Hence CICM holds for this property also . Property 9 : -LRB- P -RRB- -LRB- Q -RRB- -LRB- | P | + | Q | -RRB- < -LRB- | P ; Q | -RRB- OR -LRB- P -RRB- -LRB- Q -RRB- -LRB- R -RRB- -LRB- | P | + | Q | + | R | -RRB- < -LRB- | P ; Q ; R | -RRB- For the program illustrated in Fig. 4 , if we separate the main program body P by segregating Q -LRB- prime check -RRB- and R -LRB- factorial -RRB- , we have the program illustrated in Fig. 9 as shown in Appendix I . The above program has one sequential and one branch BCS . Thus cognitive weight is 4 and WICS is 1.475 . Therefore CICM = 4.425 . Hence 4.425 + 27.79 + 35.77 < 117.3 . This proves that CICM also holds for this property . In this section cognitive information complexity measure has been compared with other complexity measures in terms of all nine Weyuker 's properties . P.N. - Property Number , S.C. - Statement Count , C N. - Cyclomatic Number , E.M. - Effort Measure , D.C.-Dataflow Complexity , C.C.M. - Cognitive Complexity Measure , CICM Cognitive Information Complexity Measure , Y - Yes , N NO P N SC CN EM DC CCM CICM . 1 Y Y Y Y Y Y 2 Y N Y N Y Y 3 Y Y Y Y Y Y 4 Y Y Y Y Y Y 5 Y Y N N Y Y 6 N N Y Y N Y 7 N N N Y Y Y 8 Y Y Y Y Y Y 9 N N Y Y Y Y Table 1 : Comparison of complexity measures with Weyuker properties . It may be observed from the table 1 that complexity of a program using effort measure , data flow measure and Cognitive Information Complexity Measure depend directly on the placement of statement and therefore all these measures hold for property 6 also . All the complexity measure intend to rank all the programs differently Software complexity measures serves both as an analyzer and a predicator in quantitative software engineering . Software quality is defined as the completeness , correctness , consistency , no misinterpretation , and no ambiguity , feasible verifiable in both specification and implementation . For a good complexity measure it is very necessary that the particular complexity measure not only satisfies the above-mentioned property of software quality but also satisfies the nine Weyuker properties . The software complexity in terms of cognitive information complexity measure thus has been established as a well - structured complexity measure . -LSB- 1 -RSB- Misra , S and Misra , A.K. -LRB- 2005 -RRB- : Evaluating Cognitive Complexity measure with Weyuker Properties , Proceeding of the3 rd IEEE International Conference on Cognitive Informatics -LRB- ICCI ' 04 -RRB- . ACM SIGSOFT Software Engineering Notes Page 3 January 2006 Volume 31 Number 1 -LSB- 2 -RSB- Kushwaha , D.S and Misra , A.K. -LRB- 2005 -RRB- : A Modified Cognitive Information Complexity Measure of Software , Proceeding of the 7 th International Conference on Cognitive Systems -LRB- ICCS ' 05 -RRB- -LRB- accepted for presentation -RRB- -LSB- 3 -RSB- Basili , V.R. and Phillips -LRB- 1983 -RRB- : T.Y , Metric analysis and data validation across fortran projection . IEEE Trans.software Eng. , SE 9 -LRB- 6 -RRB- : 652-663,1983 . -LSB- 4 -RSB- Basili , V.R. -LRB- 1980 -RRB- : Qualitative software complexity model : A summary in tutorial on models and method for software management and engineering . IEEE Computer Society Press , Los Alamitos , CA ,1980 . -LSB- 5 -RSB- Halstead , M. -LRB- 1977 -RRB- : Elements of software science , Elsevier North Holland , New York .1997 . -LSB- 6 -RSB- Klemola , T. and Rilling , J. -LRB- 2003 -RRB- : A Cognitive Complexity Metric Based on Category Learning , , Proceeding of the 2 nd IEEE International Conference on Cognitive Informatics -LRB- ICCI ' 03 -RRB- . -LSB- 7 -RSB- Kearney , J.K. , Sedlmeyer , R. L. , Thompson , W.B. , Gray , M. A. and Adler , M. A. -LRB- 1986 -RRB- : Software complexity measurement . ACM Press , Newyork , 28:1044-1050,1986 -LSB- 8 -RSB- McCabe , T.A. -LRB- 1976 -RRB- : Complexity measure . IEEE trans.software engg. , -LRB- se-2,6 -RRB- : 308-320,1976 . -LSB- 9 -RSB- Oviedo , E. -LRB- 1980 -RRB- : Control flow , data and program complexity . in Proc . IEEE COMPSAC , Chicago , lL , pages 146-152 , November 1980 . -LSB- 10 -RSB- Wang . and Shao , J. -LRB- 2002 -RRB- : On cognitive informatics , keynote lecture , proceeding of the .1 st IEEE International Conference on Cognitive Informatics , pages 34-42 , August 2002 . -LSB- 11 -RSB- Wang , Y . and Shao , J. -LRB- 2004 -RRB- : Measurement of the Cognitive Functional Complexity of Software , Proceeding of the 3 rd IEEE International Conference on Cognitive Informatics -LRB- ICCI ' 04 -RRB- -LSB- 12 -RSB- Wang , Y . and Shao , J. -LRB- 2003 -RRB- : On cognitive informatics , Proceeding of the 2 nd IEEE International Conference on Cognitive Informatics -LRB- ICCI ' 03 -RRB- , London , England , IEEE CS Press , pages 67-71 , August 2003 . -LSB- 13 -RSB- Wang , Y. -LRB- 2002 -RRB- : The real time process algebra -LRB- rtpa -RRB- . Annals of Software Engineering , an international journal , and 14:235-247 ,2002 . -LSB- 14 -RSB- Weyuker , E. -LRB- 1988 -RRB- : Evaluating software complexity measure . IEEE Transaction on Software Complexity Measure , 14 -LRB- 9 -RRB- : 1357-1365 , september1988 . ACM SIGSOFT Software Engineering Notes Page 4 January 2006 Volume 31 Number 1 Appendix I / * Calculate the sum of first n integer * / main -LRB- -RRB- -LCB- int i , n , sum = 0 ; printf -LRB- `` enter the number '' -RRB- ; / / BCS1 scanf -LRB- '' % d '' , & n -RRB- ; for -LRB- i = 1 ; i < = n ; i + + -RRB- / / BCS2 sum = sum + i ; printf -LRB- `` the sum is % d '' , sumssss -RRB- ; getch -LRB- -RRB- ; -RCB- Fig. 1 : Source code of the sum of first n integers . main -LRB- -RRB- -LCB- int b ; int sum = 0 ; Printf -LRB- `` Enter the Number '' -RRB- ; Scanf -LRB- '' % d '' , & n -RRB- ; Sum = -LRB- b +1 -RRB- * b/2 ; Printf -LRB- `` The sum is % d '' , sum -RRB- ; getch -LRB- -RRB- ; -RCB- Fig. 2 : Source code to calculate sum of first n integers . # define N 10 main -LRB- -RRB- -LCB- int count float , sum , average , number ; sum = count = 0 ; while -LRB- count < N -RRB- -LCB- scanf -LRB- '' % f '' , & number -RRB- ; sum = sum + number ; count = count +1 ; -RCB- average = sum / N ; printf -LRB- `` Average = % f '' , average -RRB- ; -RCB- Fig. 3 : Source code to calculate the average of a set of N numbers . #include < stdio.h > #include < stdlib.h > int main -LRB- -RRB- -LCB- long fact -LRB- int n -RRB- ; int isprime -LRB- int n -RRB- ; int n ; long int temp ; clrscr -LRB- -RRB- ; printf -LRB- '' \ n input the number '' -RRB- ; / / BCS11 scanf -LRB- '' % d '' , & n -RRB- ; temp = fact -LRB- n -RRB- ; / / BCS12 -LCB- printf -LRB- '' \ n is prime '' -RRB- ; -RCB- int flag1 = isprime -LRB- n -RRB- ; / / BCS13 if -LRB- flag1 = = 1 -RRB- / / BCS14 else -LCB- printf -LRB- '' \ n is not prime '' -RRB- -RCB- ; printf -LRB- '' \ n factorial -LRB- n -RRB- = % d '' , temp -RRB- ; getch -LRB- -RRB- ; long fact -LRB- int n -RRB- -LCB- long int facto = 1 ; / / BCS21 if -LRB- n = = 0 -RRB- / / BCS22 facto = 1 ; else facto = n * fact -LRB- n-1 -RRB- ; return -LRB- facto -RRB- ; -RCB- int isprime -LRB- int n -RRB- -LCB- int flag ; / / BCS31 if -LRB- n = = 2 -RRB- flag = 1 ; / / BCS32 else for -LRB- int i = 2 ; i < n ; i + + -RRB- / / BCS33 -LCB- if -LRB- n % i = = 0 -RRB- / / BCS34 -LCB- flag = 0 ; Therefore Wc = 3 break ; -RCB- else -LCB- flag = 1 ; -RCB- -RCB- return -LRB- flag -RRB- ; -RCB- -RCB- Fig . 4 : Source code to check prime number and to calculate factorial of the number #include < stdio.h > #include < stdlib.h > #include < conio.h > int main -LRB- -RRB- -LCB- / / BCS1 int flag = 1 , n ; clrscr -LRB- -RRB- ; printf -LRB- '' \ n enter the number '' -RRB- ; scanf -LRB- '' % d '' , & n -RRB- ; if -LRB- n = = 2 -RRB- flag = 1 ; / / BCS21 else -LCB- for -LRB- int i = 2 ; i < n ; i + + -RRB- / / BCS22 if -LRB- n % i = = 0 -RRB- / / BCS23 -LCB- flag = 0 ; break ; -RCB- else -LCB- flag = 1 ; continue ; -RCB- -RCB- if -LRB- flag -RRB- / / BCS3 printf -LRB- `` the number is prime '' -RRB- ; else printf -LRB- `` the number is not prime '' -RRB- ; grtch -LRB- -RRB- ; -RCB- Fig. 5 : Source code for checking prime number #include < stdio.h > #include < stdlib.h > #include < conio.h > int main -LRB- -RRB- -LCB- long int fact = 1 ; int n ; clrscr -LRB- -RRB- ; ACM SIGSOFT Software Engineering Notes Page 5 January 2006 Volume 31 Number 1 printf -LRB- '' \ input the number '' -RRB- ; / / BCS1 scanf -LRB- '' % d '' , & n -RRB- ; if -LRB- n == 0 -RRB- / / BCS21 else for -LRB- int i = n ; i > 1 ; i -- -RRB- / / BCS22 fact = fact * i ; printf -LRB- '' \ n factorial -LRB- n -RRB- = % 1d '' , fact -RRB- ; getch -LRB- -RRB- ; -RCB- Fig. 6 : Source code for calculating factorial of a number Int main -LRB- -RRB- -LCB- long fact -LRB- int n -RRB- ; int i , n , sum = 0 ; printf -LRB- `` enter the number '' -RRB- ; scanf -LRB- '' % d '' , & n -RRB- ; temp = fact -LRB- n -RRB- ; for -LRB- i = 1 ; i < = n ; i + + -RRB- sum = sum + i ; printf -LRB- `` the sum is % d '' , sum -RRB- ; getch -LRB- -RRB- ; long fact -LRB- int n -RRB- -LCB- long int facto = 1 ; if -LRB- n == 0 -RRB- facto = 1 else facto = n * fact -LRB- n-1 -RRB- ; return -LRB- facto -RRB- ; -RCB- -RCB- Fig. 7 : Source code of sum of first n integer and factorial of n. main -LRB- -RRB- -LCB- int a , b , result ; result = a/b ; printf -LRB- the result is % d '' , result -RRB- ; int i , n , sum = 0 ; printf -LRB- `` enter the number '' -RRB- ; scanf -LRB- '' % d '' , & n -RRB- ; for -LRB- i = 1 ; i < = n ; i + + -RRB- sum = sum + i ; printf -LRB- `` the sum is % d '' , sum -RRB- ; getch -LRB- -RRB- ; -RCB- Fig. 8 : Source code of division and the sum of first n integers . int main -LRB- -RRB- -LCB- int n ; long int temp ; clrscr -LRB- -RRB- ; printf -LRB- '' \ n input the number '' -RRB- ; scanf -LRB- '' % d '' , & n -RRB- ; temp = fact -LRB- n -RRB- ; -LCB- printf -LRB- '' \ is prime '' -RRB- ; -RCB- int flag1 = isprime -LRB- n -RRB- ; if -LRB- flag1 == 1 -RRB- else -LCB- printf -LRB- '' \ n is not prime -RRB- -RCB- ; printf -LRB- '' \ n factorial -LRB- n -RRB- = % d '' , temp -RRB- ; getch -LRB- -RRB- ; -RCB- Fig. 9 : Source code of main program body of program in Fig. 4 ACM SIGSOFT Software Engineering Notes Page 6 January 2006 Volume 31 Number 1

O O O B I I I O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O B I I I O O O O O O O O O O O O O O O O B I I O B I I I O B I O B I I I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O B I I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O