Learning

Concepts

from

Large

Scale

Imbalanced

Data

Sets

Using

Support

Cluster

Machines

This

paper

considers

the

problem

of

using

Support

Vector

Machines

-LRB-

SVMs

-RRB-

to

learn

concepts

from

large

scale

imbalanced

data

sets

The

objective

of

this

paper

is

twofold

Firstly

we

investigate

the

effects

of

large

scale

and

imbalance

on

SVMs

We

highlight

the

role

of

linear

non-separability

in

this

problem

Secondly

we

develop

a

both

practical

and

theoretical

guaranteed

meta-algorithm

to

handle

the

trouble

of

scale

and

imbalance

The

approach

is

named

Support

Cluster

Machines

-LRB-

SCMs

-RRB-

It

incorporates

the

informative

and

the

representative

under-sampling

mechanisms

to

speedup

the

training

procedure

The

SCMs

differs

from

the

previous

similar

ideas

in

two

ways

-LRB-

a

-RRB-

the

theoretical

foundation

has

been

provided

and

-LRB-

b

-RRB-

the

clustering

is

performed

in

the

feature

space

rather

than

in

the

input

space

The

theoretical

analysis

not

only

provides

justification

but

also

guides

the

technical

choices

of

the

proposed

approach

Finally

experiments

on

both

the

synthetic

and

the

TRECVID

data

are

carried

out

The

results

support

the

previous

analysis

and

show

that

the

SCMs

are

efficient

and

effective

while

dealing

with

large

scale

imbalanced

data

sets

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Content

Analysis

and

Indexing

Abstracting

methods

Indexing

methods

I.

-LSB-

Pattern

Recognition

-RSB-

Design

Methodology-Classifier

design

and

evaluation

Algorithms

Theory

Experimentation

Supported

by

National

Natural

Science

Foundation

of

China

-LRB-

-RRB-

and

Chinese

National

Key

Foundation

Research

Development

Plan

-LRB-

-RRB-

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

MM

October

Santa

Barbara

California

USA

Copyright

ACM

In

the

context

of

concept

modelling

this

paper

considers

the

problem

of

how

to

make

full

use

of

the

large

scale

annotated

data

sets

In

particular

we

study

the

behaviors

of

Support

Vector

Machines

-LRB-

SVMs

-RRB-

on

large

scale

imbalanced

data

sets

not

only

because

its

solid

theoretical

foundations

but

also

for

its

empirical

success

in

various

applications

Motivation

Bridging

the

semantic

gap

has

been

becoming

the

most

challenging

problem

of

Multimedia

Information

Retrieval

-LRB-

MIR

-RRB-

Currently

there

are

mainly

two

types

of

methods

to

bridge

the

gap

-LSB-

-RSB-

The

first

one

is

relevance

feedback

which

attempts

to

capture

the

user

's

precise

needs

through

iterative

feedback

and

query

refinement

Another

promising

direction

is

concept

modelling

As

noted

by

Hauptmann

-LSB-

-RSB-

this

splits

the

semantic

gap

between

low

level

features

and

user

information

needs

into

two

hopefully

smaller

gaps

-LRB-

a

-RRB-

mapping

the

low-level

features

into

the

intermediate

semantic

concepts

and

-LRB-

b

-RRB-

mapping

these

concepts

into

user

needs

The

automated

image

annotation

methods

for

CBIR

and

the

high

level

feature

extraction

methods

in

CBVR

are

all

the

efforts

to

model

the

first

mapping

Of

these

methods

supervised

learning

is

one

of

the

most

successful

ones

An

early

difficulty

of

supervised

learning

is

the

lack

of

annotated

training

data

Currently

however

it

seems

no

longer

a

problem

This

is

due

to

both

the

techniques

developed

to

leverage

surrounding

texts

of

web

images

and

the

large

scale

collaborative

annotation

Actually

there

is

an

underway

effort

named

Large

Scale

Concept

Ontology

for

Multimedia

Understanding

-LRB-

LSCOM

-RRB-

which

intends

to

annotate

concepts

in

broadcast

news

video

-LSB-

-RSB-

The

initial

fruits

of

this

effort

have

been

harvested

in

the

practice

of

TRECVID

hosted

by

National

Institute

of

Standards

and

Technology

-LRB-

NIST

-RRB-

-LSB-

-RSB-

In

TRECVID

concepts

are

annotated

by

multiple

participants

through

web

collaboration

and

ten

of

them

are

used

in

the

evaluation

The

available

large

amount

of

annotated

data

is

undoubt-edly

beneficial

to

supervised

learning

However

it

also

brings

out

a

novel

challenge

that

is

how

to

make

full

use

of

the

data

while

training

the

classifiers

On

the

one

hand

the

annotated

data

sets

are

usually

in

rather

large

scale

The

velopment

set

of

TRECVID

includes

keyframes

The

data

set

of

LSCOM

with

over

annotated

concepts

might

be

even

larger

With

all

the

data

the

training

of

SVMs

will

be

rather

slow

On

the

other

hand

each

concept

will

be

the

minority

class

under

one-against-all

strategy

Only

a

small

portion

of

the

data

belong

to

the

concept

while

all

the

others

are

not

-LRB-

In

our

case

the

minority

class

always

refers

to

the

positive

class

-RRB-

The

ratio

of

the

positive

examples

and

the

negative

ones

is

typically

below

in

TRECVID

data

These

novel

challenges

have

spurred

great

interest

in

the

communities

of

data

mining

and

machine

learning

-LSB-

-RSB-

Our

first

motivation

is

to

investigate

the

effects

of

large

scale

and

imbalance

on

SVMs

This

is

critical

for

correct

technical

choices

and

development

The

second

objective

of

this

paper

is

to

provide

a

practical

as

well

as

theoretical

guaranteed

approach

to

addressing

the

problem

Our

Results

The

major

contribution

of

this

paper

can

be

summarized

as

follows

We

investigate

the

effects

of

large

scale

and

imbalance

on

SVMs

and

highlight

the

role

of

linear

non-separability

of

the

data

sets

We

find

that

SVMs

has

no

difficulties

with

linear

separable

large

scale

imbalanced

data

We

establish

the

relations

between

the

SVMs

trained

on

the

centroids

of

the

clusters

and

the

SVMs

obtained

on

the

original

data

set

We

show

that

the

difference

between

their

optimal

solutions

are

bounded

by

the

perturbation

of

the

kernel

matrix

We

also

prove

the

optimal

criteria

for

approximating

the

original

optimal

solutions

We

develop

a

meta-algorithm

named

Support

Cluster

Machines

-LRB-

SCMs

-RRB-

A

fast

kernel

k-means

approach

has

been

employed

to

partition

the

data

in

the

feature

space

rather

than

in

the

input

space

Experiments

on

both

the

synthetic

data

and

the

TRECVID

data

are

carried

out

The

results

support

the

previous

analysis

and

show

that

the

SCMs

are

efficient

and

effective

while

dealing

with

large

scale

imbalanced

data

sets

Organization

The

structure

of

this

paper

is

as

follows

In

Section

we

give

a

brief

review

of

SVMs

and

kernel

k-means

We

discuss

the

effects

of

the

large

scale

imbalanced

data

on

SVMs

in

Section

We

develop

the

theoretical

foundations

and

present

the

detailed

SCMs

approach

in

Section

In

Section

we

carry

out

experiments

on

both

the

synthetic

and

the

TRECVID

data

sets

Finally

we

conclude

the

paper

in

Section

Here

we

present

a

sketch

introduction

to

the

soft-margin

SVMs

for

the

convenience

of

the

deduction

in

Section

For

a

binary

classification

problem

given

a

training

data

set

D

of

size

n

D

-LCB-

-LRB-

x

i

y

i

-RRB-

x

i

R

N

y

i

-LCB-

-RCB-

-RCB-

where

x

i

indicates

the

training

vector

of

the

ith

sample

and

y

i

indicates

its

target

value

and

i

n

The

classification

hyperplane

is

defined

as

w

-LRB-

x

-RRB-

b

where

-LRB-

-RRB-

is

a

mapping

from

R

N

to

a

-LRB-

usually

-RRB-

higher

dimension

Hilbert

space

H

and

denotes

the

dot

product

in

H.

Thus

the

decision

function

f

-LRB-

x

-RRB-

is

f

-LRB-

x

-RRB-

sign

-LRB-

w

-LRB-

x

-RRB-

b

-RRB-

The

SVMs

aims

to

find

the

hyperplane

with

the

maximum

margin

between

the

two

classes

i.e.

the

optimal

hyperplane

This

can

be

obtained

by

solving

the

following

quadratic

optimization

problem

min

w

b

w

C

n

i

i

subject

to

y

i

-LRB-

w

-LRB-

x

i

-RRB-

b

-RRB-

i

-LRB-

-RRB-

i

i

n.

With

the

help

of

Lagrange

multipliers

the

dual

of

the

above

problem

is

min

G

-LRB-

-RRB-

T

Q

e

T

subject

to

i

C

i

n

-LRB-

-RRB-

T

y

where

is

a

vector

with

components

i

that

are

the

Lagrange

multipliers

C

is

the

upper

bound

e

is

a

vector

of

all

ones

and

Q

is

an

n

n

positive

semi-definite

matrix

Q

ij

y

i

y

j

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

Since

the

mapping

-LRB-

-RRB-

only

appears

in

the

dot

product

therefore

we

need

not

know

its

explicit

form

Instead

we

define

a

kernel

K

-LRB-

-RRB-

to

calculate

the

dot

product

i.e.

K

-LRB-

x

i

x

j

-RRB-

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

The

matrix

K

with

components

K

-LRB-

x

i

x

j

-RRB-

is

named

Gram

Matrix

-LRB-

or

kernel

matrix

-RRB-

With

kernel

K

we

can

implicitly

map

the

training

data

from

input

space

to

a

feature

space

H.

Kernel

k

means

and

Graph

Partitioning

Given

a

set

of

vectors

x

x

n

the

standard

k-means

algorithm

aims

to

find

clusters

k

that

minimize

the

objective

function

J

-LRB-

-LCB-

c

-RCB-

k

c

-RRB-

k

c

x

i

c

x

i

m

c

-LRB-

-RRB-

where

-LCB-

c

-RCB-

k

c

denotes

the

partitioning

of

the

data

set

and

m

c

xic

x

i

c

is

the

centroid

of

the

cluster

c

Similar

to

the

idea

of

nonlinear

SVMs

the

k-means

can

also

be

performed

in

the

feature

space

with

the

help

of

a

nonlinear

mapping

-LRB-

-RRB-

which

results

in

the

so-called

kernel

k-means

J

-LRB-

-LCB-

c

-RCB-

k

c

-RRB-

k

c

x

i

c

-LRB-

x

i

-RRB-

m

c

-LRB-

-RRB-

where

m

c

xic

-LRB-

x

i

-RRB-

c

If

we

expand

the

Euclidean

distance

-LRB-

x

i

-RRB-

m

c

in

the

objective

function

we

can

find

that

the

image

of

x

i

only

appears

in

the

form

of

dot

product

Thus

given

a

kernel

matrix

K

with

the

same

meaning

in

SVMs

we

can

compute

the

distance

between

points

and

centroids

without

knowing

explicit

representation

of

-LRB-

x

i

-RRB-

Recently

an

appealing

alternative

i.e.

the

graph

clustering

has

attracted

great

interest

It

treats

clustering

as

a

graph

partition

problem

Given

a

graph

G

-LRB-

V

E

A

-RRB-

which

consists

of

a

set

of

vertices

V

and

a

set

of

edges

E

such

that

an

edge

between

two

vertices

represents

their

similarity

The

affinity

matrix

A

is

V

V

whose

entries

represent

the

weights

of

the

edges

Let

links

-LRB-

V

V

-RRB-

be

the

sum

of

the

edge

weights

between

the

nodes

in

V

and

V

that

is

links

-LRB-

V

V

-RRB-

iV

jV

A

ij

Ratio

association

is

a

type

of

graph

partitioning

objective

which

aims

to

maximize

within-cluster

association

relative

to

the

size

of

the

cluster

RAssoc

-LRB-

G

-RRB-

max

V

V

k

k

c

links

-LRB-

V

c

V

c

-RRB-

V

c

-LRB-

-RRB-

The

following

theorem

establishes

the

relation

between

kernel

k-means

and

graph

clustering

-LSB-

-RSB-

With

this

result

we

can

develop

some

techniques

to

handle

the

difficulty

of

storing

the

large

kernel

matrix

for

kernel

k-means

Theorem

Given

a

data

set

we

can

construct

a

weighted

graph

G

-LRB-

V

E

A

-RRB-

by

treating

each

sample

as

a

node

and

linking

an

edge

between

each

other

If

we

define

the

edge

weight

A

ij

K

-LRB-

x

i

x

j

-RRB-

that

is

A

K

the

minimization

of

-LRB-

-RRB-

is

equivalent

to

the

maximization

of

-LRB-

-RRB-

There

are

two

obstacles

yielded

by

large

scale

The

first

one

is

the

kernel

evaluation

which

has

been

intensively

discussed

in

the

previous

work

The

computational

cost

scales

quadratically

with

the

data

size

Furthermore

it

is

impossible

to

store

the

whole

kernel

matrix

in

the

memory

for

common

computers

The

decomposition

algorithms

-LRB-

e.g.

SMO

-RRB-

have

been

developed

to

solve

the

problem

-LSB-

-RSB-

The

SMO-like

algorithms

actually

transform

the

space

load

to

the

time

cost

i.e.

numerous

iterations

until

convergence

To

reduce

or

avoid

the

kernel

reevaluations

various

efficient

caching

techniques

are

also

proposed

-LSB-

-RSB-

Another

obstacle

caused

by

large

scale

is

the

increased

classification

difficulty

that

is

the

more

probable

data

overlapping

We

can

not

prove

it

is

inevitable

but

it

typically

happens

Assume

we

will

draw

n

randomly

chosen

numbers

between

to

from

a

uniform

distribution

our

chances

of

drawing

a

number

close

to

would

improve

with

increasing

values

of

n

even

though

the

expected

mean

of

the

draws

is

invariant

-LSB-

-RSB-

The

checkerboard

experiment

in

-LSB-

-RSB-

is

an

intuitive

example

This

is

true

especially

for

the

real

world

data

either

because

of

the

weak

features

-LRB-

we

mean

features

that

are

less

discriminative

-RRB-

or

because

of

the

noises

With

the

large

scale

data

the

samples

in

the

overlapping

area

might

be

so

many

that

the

samples

violating

KKT

conditions

become

abundant

This

means

the

SMO

algorithm

might

need

more

iterations

to

converge

Generally

the

existing

algorithmic

approaches

have

not

been

able

to

tackle

the

very

large

data

set

Whereas

the

under-sampling

method

e.g.

active

learning

is

possible

With

unlabelled

data

active

learning

selects

a

well-chosen

subset

of

data

to

label

so

that

reduce

the

labor

of

manual

annotations

-LSB-

-RSB-

With

large

scale

labelled

data

active

learning

can

also

be

used

to

reduce

the

scale

of

training

data

-LSB-

-RSB-

The

key

issue

of

active

learning

is

how

to

choose

the

most

valuable

samples

The

informative

sampling

is

a

popular

criterion

That

is

the

samples

closest

to

the

boundary

or

maximally

violating

the

KKT

conditions

-LRB-

the

misclassified

samples

-RRB-

are

preferred

-LSB-

-RSB-

Active

learning

is

usually

in

an

iterative

style

It

requires

an

initial

-LRB-

usually

random

selected

-RRB-

data

set

to

obtain

the

estimation

of

the

boundary

The

samples

selected

in

the

following

iterations

depend

on

this

initial

boundary

In

addition

active

learning

can

not

work

like

the

decomposition

approach

which

stops

until

all

the

samples

satisfy

the

KKT

conditions

This

imply

a

potential

danger

that

is

if

the

initial

data

are

selected

improperly

the

algorithm

might

not

be

able

to

find

the

suitable

hyperplane

Thus

another

criterion

i.e.

representative

must

be

considered

Here

representative

refers

to

the

ability

to

characterize

the

data

distribution

Nguyen

et

al.

-LSB-

-RSB-

show

that

the

active

learning

method

considering

the

representative

criterion

will

achieve

better

results

Specifically

for

SVMs

pre-clustering

is

proposed

to

estimate

the

data

distribution

before

the

under-sampling

-LSB-

-RSB-

Similar

ideas

of

representative

sampling

appear

in

-LSB-

-RSB-

The

Imbalanced

Data

The

reason

why

general

machine

learning

systems

suffer

performance

loss

with

imbalanced

data

is

not

yet

clear

-LSB-

-RSB-

but

the

analysis

on

SVMs

seems

relatively

straightforward

Akbani

et

al.

have

summarized

three

possible

causes

for

SVMs

-LSB-

-RSB-

They

are

-LRB-

a

-RRB-

positive

samples

lie

further

from

the

ideal

boundary

-LRB-

b

-RRB-

the

weakness

of

the

soft-margin

SVMs

and

-LRB-

c

-RRB-

the

imbalanced

support

vector

ratio

Of

these

causes

in

our

opinion

what

really

matters

is

the

second

one

The

first

cause

is

pointed

out

by

Wu

et

al.

-LSB-

-RSB-

This

situation

occurs

when

the

data

are

linearly

separable

and

the

imbalance

is

caused

by

the

insufficient

sampling

of

the

minority

class

Only

in

this

case

does

the

ideal

boundary

make

sense

As

for

the

third

cause

Akbani

et

al.

have

pointed

out

that

it

plays

a

minor

role

because

of

the

constraint

T

y

on

Lagrange

multipliers

-LSB-

-RSB-

The

second

cause

states

that

the

soft-margin

SVMs

has

inherent

weakness

for

handling

imbalanced

data

We

find

that

it

depends

on

the

linear

separability

of

the

data

whether

the

imbalance

has

negative

effects

on

SVMs

For

linearly

separable

data

the

imbalance

will

have

tiny

effects

on

SVMs

since

all

the

slack

variables

of

-LRB-

-RRB-

tend

to

be

zeros

-LRB-

unless

the

C

is

so

small

that

the

maximization

of

the

margin

dominates

the

objective

-RRB-

In

the

result

there

is

no

contradiction

between

the

capacity

of

the

SVMs

and

the

empirical

error

Unfortunately

linear

non-separable

data

often

occurs

The

SVMs

has

to

achieve

a

tradeoff

between

maximizing

the

margin

and

minimizing

the

empirical

error

For

imbalanced

data

the

majority

class

outnumbers

the

minority

one

in

the

overlapping

area

To

reduce

the

overwhelming

errors

of

misclassifying

the

majority

class

the

optimal

hyperplane

will

inevitably

be

skew

to

the

minority

In

the

extreme

if

C

is

not

very

large

SVMs

simply

learns

to

classify

everything

as

negative

because

that

makes

the

margin

the

largest

with

zero

cumulative

error

on

the

abundant

negative

examples

The

only

tradeoff

is

the

small

amount

of

cumulative

error

on

the

few

positive

examples

which

does

not

count

for

much

Several

variants

of

SVMs

have

been

adopted

to

solve

the

problem

of

imbalance

One

choice

is

the

so-called

one-class

SVMs

which

uses

only

positive

examples

for

training

Without

using

the

information

of

the

negative

samples

it

is

usually

difficult

to

achieve

as

good

result

as

that

of

binary

SVMs

classifier

-LSB-

-RSB-

Using

different

penalty

constants

C

and

C

for

the

positive

and

negative

examples

have

been

reported

to

be

effective

-LSB-

-RSB-

However

Wu

et

al.

point

out

that

the

effectiveness

of

this

method

is

limited

-LSB-

-RSB-

The

explanation

of

Wu

is

based

on

the

KKT

condition

T

y

which

imposes

an

equal

total

influence

from

the

positive

and

negative

support

vectors

We

evaluate

this

method

and

the

result

shows

that

tuning

C

C

does

work

-LRB-

details

refer

to

Section

-RRB-

We

find

this

also

depends

on

the

linear

separability

of

the

data

whether

this

method

works

For

linearly

separable

data

tuning

C

C

has

little

effects

since

the

penalty

constants

are

useless

with

the

zero-valued

slack

variables

However

if

the

data

are

linearly

non-separable

tuning

C

C

does

change

the

position

of

separating

hyperplane

The

method

to

modify

the

kernel

matrix

is

also

proposed

to

improve

SVMs

for

imbalanced

data

-LSB-

-RSB-

A

possible

drawback

of

this

type

approach

is

its

high

computational

costs

The

proposed

approach

is

named

Support

Cluster

Machines

-LRB-

SCMs

-RRB-

We

first

partition

the

negative

samples

into

disjoint

clusters

then

train

an

initial

SVMs

model

using

the

positive

samples

and

the

representatives

of

the

negative

clusters

With

the

global

picture

of

the

initial

SVMs

we

can

approximately

identify

the

support

vectors

and

non-support

vectors

A

shrinking

technique

is

then

used

to

remove

the

samples

which

are

most

probably

not

support

vectors

This

procedure

of

clustering

and

shrinking

are

performed

itera-tively

several

times

until

some

stop

criteria

satisfied

With

such

a

from

coarse-to-fine

procedure

the

representative

and

informative

mechanisms

are

incorporated

There

are

four

key

issues

in

the

meta-algorithm

of

SCMs

-LRB-

a

-RRB-

How

to

get

the

partition

of

the

training

data

-LRB-

b

-RRB-

How

to

get

the

representative

for

each

cluster

-LRB-

c

-RRB-

How

to

safely

remove

the

non-support

vector

samples

-LRB-

d

-RRB-

When

to

stop

the

iteration

procedure

Though

similar

ideas

have

been

proposed

to

speed-up

SVMs

in

-LSB-

-RSB-

no

theoretical

analysis

of

this

idea

has

been

provided

In

the

following

we

present

an

in-depth

analysis

for

this

type

of

approaches

and

attempt

to

improve

the

algorithm

under

the

theoretical

guide

Theoretical

Analysis

Suppose

-LCB-

c

-RCB-

k

c

is

a

partition

of

the

training

set

that

the

samples

within

the

same

cluster

have

the

same

class

label

If

we

construct

a

representative

u

c

for

each

cluster

c

we

can

obtain

two

novel

models

of

SVMs

The

first

one

is

named

Support

Cluster

Machines

-LRB-

SCMs

-RRB-

It

treats

each

representative

as

a

sample

thus

the

data

size

is

reduced

from

n

to

k

This

equals

to

the

classification

of

the

clusters

That

is

where

the

name

SCMs

come

from

The

new

training

set

is

D

-LCB-

-LRB-

u

c

y

c

-RRB-

u

c

R

N

y

c

-LCB-

-RCB-

c

k

-RCB-

in

which

y

c

equals

the

labels

of

the

samples

within

c

We

define

the

dual

problem

of

support

cluster

machines

as

min

G

-LRB-

-RRB-

T

Q

e

T

subjectto

i

i

C

i

k

-LRB-

-RRB-

T

y

where

is

a

vector

of

size

k

with

components

i

corresponding

to

u

i

i

C

is

the

upper

bound

for

i

e

is

a

k

dimension

vector

of

all

ones

and

Q

is

an

k

k

positive

semi-definite

matrix

Q

ij

y

i

y

j

-LRB-

u

i

-RRB-

-LRB-

u

j

-RRB-

Another

one

is

named

Duplicate

Support

Vector

Machines

-LRB-

DSVMs

-RRB-

Different

from

SCMs

it

does

not

reduce

the

size

of

training

set

Instead

it

replace

each

sample

x

i

with

the

representative

of

the

cluster

that

x

i

belongs

to

Thus

the

samples

within

the

same

cluster

are

duplicate

That

is

why

it

is

named

DSVMs

The

training

set

is

D

-LCB-

-LRB-

x

i

y

i

-RRB-

x

i

D

if

x

i

c

x

i

u

c

and

y

i

y

i

-RCB-

and

the

corresponding

dual

problem

is

defined

as

min

G

-LRB-

-RRB-

T

Q

e

T

subjectto

i

C

i

n

-LRB-

-RRB-

T

y

where

Q

is

is

an

n

n

positive

semi-definite

matrix

Q

ij

y

i

y

j

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

We

have

the

following

theorem

that

states

-LRB-

-RRB-

is

somehow

equivalent

to

-LRB-

-RRB-

Theorem

With

the

above

definitions

of

the

SCMs

and

the

DSVMs

if

and

are

their

optimal

solutions

respectively

the

relation

G

-LRB-

-RRB-

G

-LRB-

-RRB-

holds

Furthermore

any

R

k

satisfying

-LCB-

c

x

i

c

i

c

k

-RCB-

is

the

optimal

solution

of

SCMs

Inversely

any

R

n

satisfying

-LCB-

x

i

c

i

c

c

k

-RCB-

and

the

constraints

of

-LRB-

-RRB-

is

the

optimal

solution

of

DSVMs

The

proof

is

in

Appendix

A.

Theorem

shows

that

solving

the

SCMs

is

equivalent

to

solving

a

quadratic

programming

problem

of

the

same

scale

as

that

of

the

SVMs

in

-LRB-

-RRB-

Comparing

-LRB-

-RRB-

and

-LRB-

-RRB-

we

can

find

that

only

the

Hessian

matrix

is

different

Thus

to

estimate

the

approximation

from

SCMs

of

-LRB-

-RRB-

to

SVMs

of

-LRB-

-RRB-

we

only

need

to

analyze

the

stability

of

the

quadratic

programming

model

in

-LRB-

-RRB-

when

the

Hessian

matrix

varies

from

Q

to

Q.

Daniel

has

presented

a

study

on

the

stability

of

the

solution

of

definite

quadratic

programming

which

requires

that

both

Q

and

Q

are

positive

definite

-LSB-

-RSB-

However

in

our

situation

Q

is

usually

positive

definite

and

Q

is

not

-LRB-

because

of

the

duplications

-RRB-

We

develop

a

novel

theorem

for

this

case

If

define

Q

Q

where

denotes

the

Frobenius

norm

of

a

matrix

the

value

of

measure

the

size

of

the

perturbations

between

Q

and

Q

We

have

the

following

theorem

Theorem

If

Q

is

positive

definite

and

Q

Q

let

and

be

the

optimal

solutions

to

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

we

have

mC

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-LRB-

m

m

-RRB-

C

where

is

the

minimum

eigenvalue

of

Q

m

and

m

indicate

the

numbers

of

the

support

vectors

for

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

The

proof

is

in

Appendix

B

This

theorem

shows

that

the

approximation

from

-LRB-

-RRB-

to

-LRB-

-RRB-

is

bounded

by

Note

that

this

does

not

mean

that

with

minimal

we

are

sure

to

get

the

best

approximate

solution

For

example

adopting

the

support

vectors

of

-LRB-

-RRB-

to

construct

Q

will

yield

the

exact

optimal

solution

of

-LRB-

-RRB-

but

the

corresponding

are

not

necessarily

minimum

However

we

do

not

know

which

samples

are

support

vectors

beforehand

What

we

can

do

is

to

minimize

the

potential

maximal

distortion

between

the

solutions

between

-LRB-

-RRB-

and

-LRB-

-RRB-

Now

we

consider

the

next

problem

that

is

given

the

partition

-LCB-

c

-RCB-

k

c

what

are

the

best

representatives

-LCB-

u

c

-RCB-

k

c

for

the

clusters

in

the

sense

of

approximating

Q

In

fact

we

have

the

following

theorem

Theorem

Given

the

partition

-LCB-

c

-RCB-

k

c

the

-LCB-

u

c

-RCB-

k

c

satisfying

-LRB-

u

c

-RRB-

x

i

c

-LRB-

x

i

-RRB-

c

c

k

-LRB-

-RRB-

will

make

Q

Q

minimum

The

proof

is

in

Appendix

C

This

theorem

shows

that

given

the

partition

-LRB-

u

c

-RRB-

m

c

yields

the

best

approximation

between

Q

and

Q

Here

we

come

to

the

last

question

i.e.

what

partition

-LCB-

c

-RCB-

k

c

will

make

Q

Q

minimum

To

make

the

problem

more

clearly

we

expand

as

Q

Q

k

h

k

l

x

i

h

x

j

l

-LRB-

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

m

h

m

l

-RRB-

-LRB-

-RRB-

There

are

approximately

k

n

k

types

of

such

partitions

of

the

data

set

An

exhaustive

search

for

the

best

partition

is

impossible

Recalling

that

-LRB-

-RRB-

is

similar

to

-LRB-

-RRB-

we

have

the

following

theorem

which

states

their

relaxed

equivalence

Theorem

The

relaxed

optimal

solution

of

minimizing

-LRB-

-RRB-

and

the

relaxed

optimal

solution

of

minimizing

-LRB-

-RRB-

are

equivalent

The

proof

can

be

found

in

Appendix

D.

Minimizing

amounts

to

find

a

low-rank

matrix

approximating

Q.

Ding

et

al.

have

pointed

out

the

relaxed

equivalence

between

kernel

PCA

and

kernel

k-means

in

-LSB-

-RSB-

Note

that

minimizing

-LRB-

-RRB-

is

different

from

kernel

PCA

in

that

it

is

with

an

additional

block-wise

constant

constraint

That

is

the

value

of

Q

ij

must

be

invariant

with

respect

to

the

cluster

h

containing

x

i

and

the

cluster

l

containing

x

j

With

Theorem

we

know

that

kernel

k-means

is

a

suitable

method

to

obtain

the

partition

of

data

According

to

the

above

results

the

SCMs

essentially

finds

an

approximate

solution

to

the

original

SVMs

by

smoothing

the

kernel

matrix

K

-LRB-

or

Hessian

matrix

Q

-RRB-

Fig.

illustrates

the

procedure

of

smoothing

the

kernel

matrix

via

clustering

Hence

by

solving

a

smaller

quadratic

programming

problem

the

position

of

separating

hyperplane

can

be

roughly

determined

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

-LRB-

a

-RRB-

data

distribution

-LRB-

b

-RRB-

the

visualization

of

the

kernel

matrix

Q

-LRB-

c

-RRB-

the

kernel

matrix

Q

by

re-ordering

the

entries

so

that

the

samples

belonging

to

the

same

cluster

come

together

-LRB-

d

-RRB-

the

approximate

kernel

matrix

Q

obtained

by

replacing

each

sample

with

the

corresponding

centroid

Kernel-based

Graph

Clustering

In

the

previous

work

k-means

-LSB-

-RSB-

BIRCH

-LSB-

-RSB-

and

PDDP

-LSB-

-RSB-

have

been

used

to

obtain

the

partition

of

the

data

of

them

performs

clustering

in

the

feature

space

though

the

SVMs

works

in

the

feature

space

This

is

somewhat

unnatural

Firstly

recalling

that

the

kernel

K

-LRB-

-RRB-

usually

implies

an

implicitly

nonlinear

mapping

from

the

input

space

to

the

feature

space

the

optimal

partition

of

input

space

is

not

necessarily

the

optimal

one

of

feature

space

Take

k-means

as

an

example

due

to

the

fact

that

the

squared

Euclidean

distance

is

used

as

the

distortion

measure

the

clusters

must

be

separated

by

piece-wise

hyperplanes

-LRB-

i.e.

voronoi

diagram

-RRB-

However

these

separating

hyperplanes

are

no

longer

hyperplanes

in

the

feature

space

with

nonlinear

mapping

-LRB-

-RRB-

Secondly

the

k-means

approach

can

not

capture

the

complex

structure

of

data

As

shown

in

Fig.

the

negative

class

is

in

a

ring-shape

in

the

input

space

If

the

k-means

is

used

the

centroids

of

positive

and

negative

class

might

overlap

Whereas

in

the

feature

space

the

kernel

k-means

might

get

separable

centroids

Several

factors

limit

the

application

of

kernel

k-means

to

large

scale

data

Firstly

it

is

almost

impossible

to

store

the

whole

kernel

matrix

K

in

the

memory

e.g.

for

n

we

still

need

gigabytes

memory

taking

the

symmetry

into

account

Secondly

the

kernel

k-means

relies

heavily

on

an

effective

initialization

to

achieve

good

results

and

we

do

not

have

such

a

sound

method

yet

Finally

the

computational

cost

of

the

kernel

k-means

might

exceeds

that

of

SVMs

and

therefore

we

lose

the

benefits

of

under-sampling

Dhillon

et

al.

recently

propose

a

multilevel

kernel

k-means

method

-LSB-

-RSB-

which

seems

to

cater

to

our

requirements

The

approach

is

based

on

the

equivalence

between

graph

clustering

and

kernel

k-means

It

incorporates

the

coarsening

and

initial

partitioning

phases

to

obtain

a

good

initial

clustering

Most

importantly

the

approach

is

extremely

efficient

It

can

handle

a

graph

with

nodes

and

edges

in

several

seconds

Therefore

here

we

adopt

this

approach

The

detailed

description

can

be

found

in

-LSB-

-RSB-

In

the

following

we

focus

on

how

to

address

the

difficulty

of

storing

large

scale

kernel

matrix

Theorem

states

that

kernel

k-means

is

equivalent

to

a

type

of

graph

clustering

Kernel

k-means

focuses

on

grouping

data

so

that

their

average

distance

from

the

centroid

is

minimum

while

graph

clustering

aims

to

minimizing

the

average

pair-wise

distance

among

the

data

Central

grouping

and

pair-wise

grouping

are

two

different

views

of

the

same

approach

From

the

perspective

of

pair-wise

grouping

we

can

expect

that

two

samples

with

large

distance

will

not

belong

to

the

same

cluster

in

the

optimal

solution

Thus

x

x

z

z

z

Positive

Class

Negative

Class

Figure

The

left

and

right

figures

show

the

data

distribution

of

input

space

and

feature

space

respectively

The

two

classes

are

indicated

by

squares

and

circles

Each

class

is

grouped

into

one

cluster

and

the

solid

mark

indicates

the

centroid

of

the

class

we

add

the

constraint

that

two

samples

with

distance

large

enough

are

not

linked

by

an

edge

that

is

transforming

the

dense

graph

to

a

sparse

graph

This

procedure

is

the

common

practice

in

spectral

clustering

or

manifold

embedding

Usually

two

methods

have

been

widely

used

for

this

purpose

i.e.

k-nearest

neighbor

and

ball

Here

we

adopt

the

ball

approach

Concretely

the

edges

with

weight

A

ij

is

removed

from

the

original

graph

in

which

the

parameter

is

pre-determined

By

transforming

a

dense

graph

into

a

sparse

graph

we

only

need

store

the

sparse

affinity

matrix

instead

of

the

original

kernel

matrix

Nevertheless

we

have

to

point

out

that

the

time

complexity

of

constructing

sparse

graph

is

O

-LRB-

n

-RRB-

for

data

set

with

n

examples

which

is

the

efficiency

bottleneck

of

the

current

implementation

With

the

sparse

graph

each

iteration

of

the

multilevel

kernel

k-means

costs

O

-LRB-

ln

-RRB-

time

where

ln

is

the

number

of

nonzero

entries

in

the

kernel

matrix

Support

Cluster

Machines

According

to

Theorem

choosing

the

centroid

of

each

cluster

as

representative

will

yield

the

best

approximation

However

the

explicit

form

of

-LRB-

-RRB-

is

unknown

We

do

n't

know

the

exact

pre-images

of

-LCB-

m

c

-RCB-

k

c

what

we

can

get

are

the

dot

products

between

the

centroids

by

m

h

m

l

h

l

x

i

h

x

j

l

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

which

requires

O

-LRB-

n

-RRB-

costs

Then

the

pre-computed

kernel

SVMs

can

be

used

The

pre-computed

kernel

SVMs

takes

the

kernel

matrix

K

as

input

and

save

the

indices

of

support

vectors

in

the

model

-LSB-

-RSB-

To

classify

the

incoming

sample

x

we

have

to

calculate

the

dot

product

between

x

and

all

the

samples

in

the

support

clusters

e.g.

c

-LRB-

If

m

c

is

a

support

vector

we

define

the

cluster

c

as

support

cluster

-RRB-

x

m

c

c

x

i

c

x

x

i

We

need

another

O

-LRB-

nm

-RRB-

costs

to

predict

all

the

training

samples

if

there

are

m

samples

in

support

clusters

This

is

unacceptable

for

large

scale

data

To

reduce

the

kernel

reevaluation

we

adopt

the

same

method

as

-LSB-

-RSB-

i.e.

selecting

a

pseudo-center

for

each

cluster

as

the

representative

u

c

arg

min

x

i

c

-LRB-

x

i

-RRB-

c

x

j

c

-LRB-

x

j

-RRB-

x

Positive

class

Negative

class

x

x

Positive

class

Negative

class

x

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

Each

class

is

grouped

into

one

cluster

-LRB-

b

-RRB-

each

class

is

grouped

into

two

clusters

The

solid

mark

represents

the

centroid

of

the

corresponding

class

The

solid

lines

indicate

the

support

hyperplanes

yielded

by

SCMs

and

the

dot

lines

indicate

the

true

support

hyperplanes

which

can

be

directly

obtained

by

u

c

arg

max

x

i

c

x

j

c

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

-LRB-

-RRB-

Thus

the

kernel

evaluation

within

training

procedure

requires

O

-LRB-

k

c

c

k

-RRB-

time

which

be

further

reduced

by

probabilistic

speedup

proposed

by

Smola

-LSB-

-RSB-

The

kernel

evaluation

of

predicting

the

training

samples

is

reduced

from

O

-LRB-

nm

-RRB-

to

O

-LRB-

ns

-RRB-

where

s

indicates

the

number

of

support

clusters

Shrinking

Techniques

With

the

initial

SCMs

we

can

remove

the

samples

that

are

not

likely

support

vectors

However

there

is

no

theoretical

guarantee

for

the

security

of

the

shrinking

In

Fig.

we

give

a

simple

example

to

show

that

the

shrinking

might

not

be

safe

In

the

example

if

the

samples

outside

the

margin

between

support

hyperplanes

are

to

be

removed

the

case

-LRB-

a

-RRB-

will

remove

the

true

support

vectors

while

the

case

-LRB-

b

-RRB-

will

not

The

example

shows

that

the

security

depends

on

whether

the

hyperplane

of

SCMs

is

parallel

to

the

true

separating

hyperplane

However

we

do

not

know

the

direction

of

true

separating

hyperplane

before

the

classification

Therefore

what

we

can

do

is

to

adopt

sufficient

initial

cluster

numbers

so

that

the

solution

of

SCMs

can

approximate

the

original

optimal

solution

enough

Specifically

for

large

scale

imbalanced

data

the

samples

satisfying

the

following

condition

will

be

removed

from

the

training

set

w

-LRB-

x

-RRB-

b

-LRB-

-RRB-

where

is

a

predefined

parameter

The

Algorithm

Yu

-LSB-

-RSB-

and

Boley

-LSB-

-RSB-

have

adopted

different

stop

criteria

In

Yu

et

al.

's

approach

the

algorithm

stops

when

each

cluster

has

only

one

sample

Whereas

Boley

et

al.

limit

the

maximum

iterations

by

a

fixed

parameter

Here

we

propose

two

novel

criteria

especially

suitable

for

imbalanced

data

The

first

one

is

to

stop

whenever

the

ratio

of

positive

and

negative

samples

is

relatively

imbalanced

Another

choice

is

the

Neyman-Pearson

criterion

that

is

minimizing

the

total

error

rate

subject

to

a

constraint

that

the

miss

rate

of

positive

class

is

less

than

some

threshold

Thus

once

the

miss

rate

of

positive

class

exceeds

some

threshold

we

stop

the

algorithm

The

overall

approach

is

illustrated

in

Algorithm

With

large

scale

balanced

data

we

carry

out

the

data

clustering

for

both

classes

separately

Whereas

with

imbalanced

data

the

clustering

and

shrinking

will

only

be

conducted

on

the

majority

class

The

computation

complexity

is

dominated

by

kernel

evaluation

Therefore

it

will

not

exceed

O

-LRB-

-LRB-

n

-RRB-

-LRB-

n

-RRB-

-RRB-

where

n

and

n

indicate

the

number

of

negative

and

positive

examples

respectively

Algorithm

Support

Cluster

Machines

Input

Training

data

set

D

D

D

Output

Decision

function

f

repeat

-LCB-

c

m

c

-RCB-

k

c

KernelKMeans

-LRB-

D

-RRB-

-LCB-

c

m

c

-RCB-

k

c

KernelKMeans

-LRB-

D

-RRB-

D

-LCB-

m

c

-RCB-

k

c

-LCB-

m

c

-RCB-

k

c

f

SVMTrain

-LRB-

D

-RRB-

f

-LRB-

D

-RRB-

SVMPredict

-LRB-

f

D

-RRB-

D

D

D

Shrinking

-LRB-

f

-LRB-

D

-RRB-

-RRB-

until

stop

criterion

is

true

The

experiments

on

both

the

synthetic

and

the

TRECVID

data

are

carried

out

The

experiments

on

synthetic

data

are

used

to

analyze

the

effects

of

large

scale

and

imbalance

on

SVMs

and

the

experiments

on

TRECVID

data

serve

to

evaluate

the

effectiveness

and

efficiency

of

SCMs

The

multilevel

kernel

graph

partitioning

code

graclus

-LSB-

-RSB-

is

adopted

for

data

clustering

and

the

well-known

LibSVM

software

-LSB-

-RSB-

is

used

in

our

experiments

All

our

experiments

are

done

in

a

Pentium

GHz

machine

with

memory

Synthetic

Data

Set

We

generate

two-dimensional

data

for

the

convenience

of

observation

Let

x

is

a

random

variable

uniformly

distributed

in

-LSB-

-RSB-

The

data

are

generated

by

D

-LCB-

-LRB-

x

y

-RRB-

y

sin

-LRB-

x

-RRB-

-LSB-

rand

-LRB-

-RRB-

-RSB-

x

-LSB-

-RSB-

-RCB-

D

-LCB-

-LRB-

x

y

-RRB-

y

sin

-LRB-

x

-RRB-

rand

-LRB-

-RRB-

x

-LSB-

-RSB-

-RCB-

where

rand

-LRB-

-RRB-

generates

the

random

numbers

uniformly

distributed

between

and

and

is

a

parameter

controlling

the

overlapping

ratio

of

the

two

classes

Fig.

and

Fig.

show

some

examples

of

the

synthetic

data

We

use

the

linear

kernel

function

in

all

the

experiments

on

synthetic

data

The

Effects

of

Scale

We

generate

two

types

of

balanced

data

i.e.

n

n

but

one

-LRB-

D

D

-LRB-

-RRB-

-RRB-

is

linearly

separable

and

the

Table

The

effects

of

scale

and

overlapping

on

the

time

costs

of

training

SVMs

-LRB-

in

seconds

-RRB-

n

n

2000 4000 8000 20000

time

-LRB-

D

-RRB-

time

-LRB-

D

-RRB-

Positive

class

Negative

class

Positive

class

Negative

class

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

example

of

non-overlapped

balanced

data

sets

-LRB-

b

-RRB-

example

of

overlapped

balanced

data

sets

Positive

class

Negative

class

Positive

class

Negative

class

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

example

of

non-overlapped

imbalanced

data

sets

-LRB-

b

-RRB-

example

of

overlapped

imbalanced

data

sets

other

-LRB-

D

D

-LRB-

-RRB-

-RRB-

is

not

as

shown

in

Fig.

We

observe

the

difference

of

the

behaviors

of

time

costs

for

D

and

D

when

the

scale

increases

With

the

same

parameter

settings

the

time

costs

of

optimizing

the

objective

for

D

and

D

are

shown

in

Table

from

which

we

can

get

two

conclusions

-LRB-

a

-RRB-

time

costs

increase

with

the

scale

and

-LRB-

b

-RRB-

in

the

same

scale

the

linearly

non-separable

data

will

cost

more

time

to

converge

The

Effects

of

Imbalance

We

generate

two

types

of

imbalanced

data

i.e.

n

n

but

one

-LRB-

D

D

-LRB-

-RRB-

-RRB-

is

linearly

separable

and

the

other

-LRB-

D

D

-LRB-

-RRB-

-RRB-

is

not

as

shown

in

Fig.

We

observe

the

difference

of

the

effects

of

imbalance

for

linearly

separable

data

D

and

linearly

non-separable

D

For

the

space

limitation

we

will

not

describe

the

detailed

results

here

but

only

present

the

major

conclusions

For

linearly

separable

data

SVMs

can

find

the

non-skew

hyperplane

if

C

is

not

too

small

In

this

situation

tuning

C

C

is

meaningless

For

linearly

non-separable

data

the

boundary

will

be

skew

to

positive

class

if

C

C

In

this

case

increasing

C

C

dose

push

the

skewed

separating

hyperplane

to

the

negative

class

For

both

D

and

D

if

the

C

is

too

small

underfitting

occurs

that

is

the

SVMs

simply

classify

all

the

samples

into

negative

class

TRECVID

Data

Set

Experimental

Setup

In

this

section

we

evaluate

the

proposed

approach

on

the

high

level

feature

extraction

task

of

TRECVID

-LSB-

-RSB-

Four

concepts

including

car

maps

sports

and

waterscape

are

chosen

to

model

from

the

data

sets

The

development

data

of

TRECVID

are

employed

and

divided

into

training

set

and

validation

set

in

equal

size

The

detailed

Table

The

details

of

the

training

set

and

validation

set

of

TRECVID

Concept

D

train

D

val

Positive

Negative

Positive

Negative

Car

Maps

Sports

Waterscape

tics

of

the

data

is

summarized

in

Table

In

our

experiments

the

global

color

autocorrelogram

feature

is

used

to

represent

the

visual

content

of

each

image

Conforming

to

the

convention

of

TRECVID

average

precision

-LRB-

AP

-RRB-

is

chosen

as

the

evaluation

criterion

Totally

five

algorithms

have

been

implemented

for

comparison

Whole

All

the

negative

examples

are

used

Random

Random

sampling

of

the

negative

examples

Active

Active

sampling

of

the

negative

examples

SCMs

I

SCMs

with

k-means

in

the

input

space

SCMs

SCMs

with

kernel

k-means

In

the

Active

method

we

firstly

randomly

select

a

subset

of

negative

examples

With

this

initial

set

we

train

an

SVMs

model

and

use

this

model

to

classify

the

whole

training

data

set

Then

the

maximally

misclassified

negative

examples

are

added

to

the

training

set

This

procedure

iterates

until

the

ratio

between

the

negative

and

the

positive

examples

exceeding

five

Since

both

the

Random

and

Active

methods

depend

on

the

initial

random

chosen

data

set

we

repeat

each

of

them

for

ten

times

and

calculate

their

average

performances

for

comparison

Both

SCMs

I

and

SCMs

methods

adopt

the

Gaussian

kernel

during

the

SVMs

classification

The

only

difference

is

that

SCMs

I

performs

data

clustering

with

k-means

in

the

input

space

while

SCMs

with

k-means

in

the

feature

space

Parameter

Settings

Currently

the

experiments

focus

on

the

comparative

performance

between

the

different

approaches

based

on

the

the

same

parameter

settings

Therefore

some

of

the

parameters

are

heuristically

determined

and

might

not

be

optimal

The

current

implementation

of

SCMs

involves

the

following

parameter

settings

-LRB-

a

-RRB-

Gaussian

kernel

is

adopted

and

the

parameters

are

selected

via

cross-validation

furthermore

the

kernel

function

of

kernel

k-means

clustering

is

adopted

the

same

as

that

of

SVMs

-LRB-

b

-RRB-

the

threshold

for

transforming

dense

graphs

to

sparse

ones

is

experimentally

determined

as

-LRB-

c

-RRB-

the

parameter

of

shrinking

technique

is

experimentally

chosen

as

-LRB-

d

-RRB-

for

SCMs

the

data

are

imbalanced

for

each

concept

we

only

carry

out

data

clustering

for

negative

classes

therefore

k

always

equals

D

and

k

is

always

chosen

as

D

-LRB-

e

-RRB-

we

stop

the

iteration

of

SCMs

when

the

number

of

the

negative

examples

are

not

more

than

the

five

times

of

that

of

the

positive

examples

Experiment

Results

The

average

performance

and

time

costs

of

the

various

approaches

are

in

Table

and

Table

respectively

We

can

see

that

both

the

Random

and

Active

methods

use

fewer

time

than

the

others

but

their

performances

are

not

as

good

as

the

others

Furthermore

the

SCMs

achieves

Table

The

average

performance

of

the

approaches

on

the

chosen

concepts

measured

by

average

precision

Concept

Whole

Random

Active

SCMs

I

SCMs

Car

Maps

Sports

Waterscape

Table

The

average

time

costs

of

the

approaches

on

the

chosen

concepts

-LRB-

in

seconds

-RRB-

Concept

Whole

Random

Active

SCMs

I

SCMs

Car

Maps

Sports

Waterscape

the

comparable

performance

with

that

of

Whole

while

uses

fewer

time

costs

Note

that

SCMs

I

also

achieves

satisfying

results

This

might

be

due

to

the

Gaussian

kernels

in

which

e

x-y

is

monotonic

with

x

y

Therefore

the

order

of

the

pair-wise

distances

is

the

same

for

both

the

input

space

and

feature

space

which

perhaps

leads

to

similar

clustering

results

In

this

paper

we

have

investigated

the

effects

of

scale

and

imbalance

on

SVMs

We

highlight

the

role

of

data

overlapping

in

this

problem

and

find

that

SVMs

has

no

difficulties

with

linear

separable

large

scale

imbalanced

data

We

propose

a

meta-algorithm

named

Support

Cluster

Machines

-LRB-

SCMs

-RRB-

for

effectively

learning

from

large

scale

and

imbalanced

data

sets

Different

from

the

previous

work

we

develop

the

theoretical

justifications

for

the

idea

and

choose

the

technical

component

guided

by

the

theoretical

results

Finally

experiments

on

both

the

synthetic

and

the

TRECVID

data

are

carried

out

The

results

support

the

previous

analysis

and

show

that

the

SCMs

are

efficient

and

effective

while

dealing

with

large

scale

imbalanced

data

sets

However

as

a

pilot

study

there

is

still

some

room

for

improvement

Firstly

we

have

not

incorporated

the

caching

techniques

to

avoid

the

kernel

reevaluations

Therefore

we

have

to

recalculate

the

dot

product

on

line

whenever

it

is

re-quired

Secondly

the

parameters

within

the

algorithms

are

currently

selected

heuristically

which

depend

on

the

tradeoff

of

efficiency

and

accuracy

We

would

like

to

thank

the

anonymous

reviewers

for

their

insightful

suggestions

We

also

thank

Dr.

Chih-Jen

Lin

for

the

code

of

libSVM

Brian

J.

Kulis

for

the

code

of

graclus

and

National

Institute

of

Standards

and

Technology

for

providing

the

TRECVID

data

sets

Finally

special

thanks

go

to

Dr.

Ya-xiang

Yuan

for

his

helpful

discussions

on

optimization

theory

-LSB-

-RSB-

TREC

Video

Retrieval

National

Institute

of

Standards

and

Technology

-LSB-

-RSB-

R.

Akbani

S.

Kwek

and

N.

Japkowicz

Applying

Support

Vector

Machines

to

Imbalanced

Datasets

In

Proceedings

of

ECML

pages

-LSB-

-RSB-

D.

Boley

and

D.

Cao

Training

Support

Vector

Machine

using

Adaptive

Clustering

In

Proceeding

of

SIAM

International

Conference

on

Data

Mining

April

-LSB-

-RSB-

S.

Boyd

and

L.

Vandenberghe

Convex

Optimization

Cambridge

University

Press

New

York

NY

USA

-LSB-

-RSB-

K.

Brinker

Incorporating

Diversity

in

Active

Learning

with

Support

Vector

Machines

In

Proceedings

of

ICML

pages

-LSB-

-RSB-

N.

V.

Chawla

N.

Japkowicz

and

A.

Kotcz

Editorial

Special

Issue

on

Learning

from

Imbalanced

Data

Sets

SIGKDD

Explor

Newsl.

-LRB-

-RRB-

-LSB-

-RSB-

J.

W.

Daniel

Stability

of

the

Solution

of

Definite

Quadratic

Programs

Mathematical

Programming

-LRB-

-RRB-

December

-LSB-

-RSB-

R.

Datta

J.

Li

and

J.

Z.

Wang

Content-based

Image

Retrieval

Approaches

and

Trends

of

the

New

Age

In

Proceedings

of

ACM

SIGMM

workshop

on

MIR

pages

-LSB-

-RSB-

I.

Dhillon

Y.

Guan

and

B.

Kulis

A

Fast

Kernel-based

Multilevel

Algorithm

for

Graph

Clustering

In

Proceeding

of

ACM

SIGKDD

pages

-LSB-

-RSB-

I.

S.

Dhillon

Y.

Guan

and

B.

Kulis

A

Unified

View

of

Graph

Partitioning

and

Weighted

Kernel

k-means

Technical

Report

The

University

of

Texas

at

Austin

Department

of

Computer

Sciences

June

-LSB-

-RSB-

C.

Ding

and

X.

He

K-means

clustering

via

principal

component

analysis

In

Proceedings

of

ICML

pages

-LSB-

-RSB-

K.-S

Goh

E.

Y.

Chang

and

W.-C

Lai

Multimodal

Concept-dependent

Active

Learning

for

Image

Retrieval

In

Proceedings

of

ACM

MM

pages

-LSB-

-RSB-

A.

G.

Hauptmann

Towards

a

Large

Scale

Concept

Ontology

for

Broadcast

Video

In

Proceedings

of

CIVR

pages

-LSB-

-RSB-

A.

G.

Hauptmann

Lessons

for

the

Future

from

a

Decade

of

Informedia

Video

Analysis

Research

In

Proceedings

of

CIVR

pages

-LSB-

-RSB-

C.-W

Hsu

C.-C

Chang

and

C.-J

Lin

A

Practical

Guide

to

Support

Vector

Classification

available

at

-LSB-

-RSB-

T.

Joachims

Making

Large-scale

Support

Vector

Machine

Learning

Practical

Advances

in

kernel

methods

support

vector

learning

pages

-LSB-

-RSB-

Y.

Lin

Y.

Lee

and

G.

Wahba

Support

Vector

Machines

for

Classification

in

Nonstandard

Situations

Machine

Learning

-LRB-

-RRB-

-LSB-

-RSB-

L.

M.

Manevitz

and

M.

Yousef

One-class

SVMs

for

Document

Classification

Journal

of

Machine

Learning

Research

-LSB-

-RSB-

H.

T.

Nguyen

and

A.

Smeulders

Active

Learning

Using

Pre-clustering

In

Proceedings

of

ICML

pages

-LSB-

-RSB-

E.

Osuna

R.

Freund

and

F.

Girosi

An

Improved

Training

Algorithm

for

Support

Vector

Machines

In

IEEE

Workshop

on

Neural

Networks

and

Signal

Processing

September

-LSB-

-RSB-

D.

Pavlov

J.

Mao

and

B.

Dom

Scaling-Up

Support

Vector

Machines

Using

Boosting

Algorithm

In

Proceeding

of

ICPR

volume

pages

-LSB-

-RSB-

J.

C.

Platt

Fast

Training

of

Support

Vector

Machines

using

Sequential

Minimal

Optimization

Advances

in

kernel

methods

support

vector

learning

pages

-LSB-

-RSB-

R.

C.

Prati

G.

E.

A.

P.

A.

Batista

and

M.

C.

Monard

Class

Imbalances

versus

Class

Overlapping

an

Analysis

of

a

Learning

System

Behavior

In

Proceedings

of

the

MICAI

pages

-LSB-

-RSB-

G.

Schohn

and

D.

Cohn

Less

is

More

Active

Learning

with

Support

Vector

Machines

In

Proceddings

of

ICML

pages

-LSB-

-RSB-

A.

J.

Smola

and

B.

Sch

okopf

Sparse

Greedy

Matrix

Approximation

for

Machine

Learning

In

Proceedings

of

ICML

pages

-LSB-

-RSB-

S.

Tong

and

E.

Chang

Support

Vector

Machine

Active

Learning

for

Image

Retrieval

In

Proceedings

of

ACM

MM

pages

-LSB-

-RSB-

K.

Veropoulos

N.

Cristianini

and

C.

Campbell

Controlling

the

Sensitivity

of

Support

Vector

Machines

In

Proceedings

of

IJCAI

-LSB-

-RSB-

G.

M.

Weiss

and

F.

J.

Provost

Learning

When

Training

Data

are

Costly

The

Effect

of

Class

Distribution

on

Tree

Induction

Journal

of

Artificial

Intelligence

Research

-LRB-

JAIR

-RRB-

-LSB-

-RSB-

G.

Wu

and

E.

Y.

Chang

KBA

Kernel

Boundary

Alignment

Considering

Imbalanced

Data

Distribution

IEEE

Transactions

on

Knowledge

and

Data

Engineering

-LRB-

-RRB-

-LSB-

-RSB-

Z.

Xu

K.

Yu

V.

Tresp

X.

Xu

and

J.

Wang

Representative

Sampling

for

Text

Classification

Using

Support

Vector

Machines

In

Proceedings

of

ECIR

pages

-LSB-

-RSB-

H.

Yu

J.

Yang

J.

Han

and

X.

Li

Making

SVMs

Scalable

to

Large

Data

Sets

using

Hierarchical

Cluster

Indexing

Data

Min

Knowl

Discov.

-LRB-

-RRB-

APPENDIX

A.

PROOF

OF

THEOREM

Firstly

we

define

which

satisfies

c

x

i

c

i

c

k

It

is

easy

to

verify

that

is

a

feasible

solution

of

SCMs

Secondly

we

define

satisfying

i

c

c

if

x

i

c

i

n

It

is

easy

to

verify

that

is

a

feasible

solution

of

DSVMs

According

to

the

relation

of

D

and

D

we

can

obtain

the

following

equation

n

i

n

j

i

y

i

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

j

y

j

n

i

i

k

h

k

l

h

y

h

-LRB-

u

h

-RRB-

-LRB-

u

l

-RRB-

l

y

l

k

h

h

which

means

G

-LRB-

-RRB-

G

-LRB-

-RRB-

Similarly

we

can

get

G

-LRB-

-RRB-

G

-LRB-

-RRB-

Using

the

fact

that

and

are

the

optimal

tions

to

SCMs

and

DSVMs

respectively

we

have

G

-LRB-

-RRB-

G

-LRB-

-RRB-

and

G

-LRB-

-RRB-

G

-LRB-

-RRB-

Thus

the

equation

G

-LRB-

-RRB-

G

-LRB-

-RRB-

holds

For

any

R

k

satisfying

-LCB-

c

x

i

c

i

c

k

-RCB-

we

know

it

is

a

feasible

solution

to

SCMs

and

G

-LRB-

-RRB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

holds

which

means

is

the

optimal

solution

of

SCMs

Similarly

for

any

R

n

satisfying

-LCB-

x

i

c

i

c

c

k

-RCB-

and

the

constraints

of

-LRB-

-RRB-

we

have

G

-LRB-

-RRB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

which

means

is

the

optimal

solution

of

DSVMs

B.

PROOF

OF

THEOREM

Note

that

the

feasible

regions

of

-LRB-

-RRB-

and

-LRB-

-RRB-

are

the

same

By

the

fact

that

and

are

optimal

solutions

to

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

we

know

that

-LRB-

-RRB-

T

G

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

T

G

-LRB-

-RRB-

-LRB-

-RRB-

hold

where

the

gradients

G

-LRB-

-RRB-

Q

e

and

G

-LRB-

-RRB-

Q

e.

-LRB-

-RRB-

Adding

-LRB-

-RRB-

and

-LRB-

-RRB-

and

then

a

little

arrangement

yields

-LRB-

-RRB-

T

-LSB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

T

-LSB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-RSB-

Substituting

-LRB-

-RRB-

in

the

above

inequality

we

get

-LRB-

-RRB-

T

Q

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

-LRB-

-RRB-

Adding

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

-LRB-

-RRB-

to

the

both

sides

of

-LRB-

-RRB-

we

have

-LRB-

-RRB-

T

Q

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

-LRB-

-RRB-

If

is

the

smallest

eigenvalue

of

Q

we

have

-LRB-

-RRB-

T

Q

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

Q

Q

and

mC

Using

-LRB-

-RRB-

we

get

mC

Now

we

turn

to

prove

the

second

result

is

the

optimal

solution

of

-LRB-

-RRB-

therefore

G

-LRB-

-RRB-

G

-LRB-

-RRB-

is

obvious

Meanwhile

we

have

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

G

-LRB-

-RRB-

G

-LRB-

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

-LRB-

-RRB-

T

-LRB-

Q

Q

-RRB-

Q

Q

Q

Q

-LRB-

m

m

-RRB-

C

C.

PROOF

OF

THEOREM

Expanding

to

be

the

explicit

function

of

-LCB-

-LRB-

u

c

-RRB-

-RCB-

k

c

we

get

YKY

Y

K

Y

in

which

Y

and

Y

denote

diagonal

matrices

whose

diagonal

elements

are

y

y

n

and

y

y

n

respectively

Using

the

fact

that

Y

equals

to

Y

we

have

Y

-LRB-

K

K

-RRB-

Y

Since

Y

only

change

the

signs

of

the

elements

of

K

K

by

Y

-LRB-

K

K

-RRB-

Y

we

have

K

K

k

h

k

l

x

i

h

x

j

l

-LRB-

-LRB-

x

i

-RRB-

-LRB-

x

j

-RRB-

-LRB-

u

h

-RRB-

-LRB-

u

l

-RRB-

-RRB-

It

is

a

biquadratic

function

of

-LCB-

-LRB-

u

c

-RRB-

-RCB-

k

c

Therefore

this

is

an

unconstrained

convex

optimization

problem

-LSB-

-RSB-

The

necessary

and

sufficient

condition

for

-LCB-

u

c

-RCB-

k

c

to

be

optimal

is

-LRB-

-LCB-

-LRB-

u

c

-RRB-

-RCB-

k

c

-RRB-

We

can

verify

that

-LRB-

u

c

-RRB-

xic

-LRB-

x

i

-RRB-

c

c

k

satisfies

the

condition

that

the

gradient

is

zero

D.

PROOF

OF

THEOREM

We

define

a

n

k

matrix

Z

as

Z

ic

c

if

x

i

c

otherwise

We

can

see

that

Z

captures

the

disjoint

cluster

memberships

There

is

only

one

non-zero

entry

in

each

row

of

Z

and

Z

T

Z

I

k

holds

-LRB-

I

k

indicates

the

identity

matrix

-RRB-

Suppose

is

the

matrix

of

the

images

of

the

samples

in

feature

space

i.e.

-LSB-

-LRB-

x

-RRB-

-LRB-

x

n

-RRB-

-RSB-

We

can

verify

that

the

matrix

ZZ

T

consists

of

the

mean

vectors

of

the

clusters

containing

the

corresponding

sample

Thus

the

can

be

written

as

Q

Q

T

-LRB-

ZZ

T

-RRB-

T

ZZ

T

Using

the

fact

that

trace

-LRB-

A

T

A

-RRB-

A

F

trace

-LRB-

A

B

-RRB-

trace

-LRB-

A

-RRB-

trace

-LRB-

B

-RRB-

and

trace

-LRB-

AB

-RRB-

trace

-LRB-

BA

-RRB-

we

have

trace

-LRB-

-LRB-

T

-RRB-

T

T

-LRB-

Z

T

T

Z

-RRB-

-LRB-

Z

T

T

Z

-RRB-

-RRB-

Since

trace

-LRB-

-LRB-

T

-RRB-

T

T

-RRB-

is

constant

minimizing

is

equivalent

to

maximizing

J

trace

-LRB-

-LRB-

Z

T

T

Z

-RRB-

-LRB-

Z

T

T

Z

-RRB-

-RRB-

-LRB-

-RRB-

With

similar

procedure

we

can

see

that

minimizing

J

-LRB-

-LCB-

c

-RCB-

k

c

-RRB-

amounts

to

maximizing

J

trace

-LRB-

Z

T

T

Z

-RRB-

-LRB-

-RRB-

Matrix

K

T

is

a

symmetric

matrix

Let

n

denote

its

eigenvalues

and

-LRB-

v

v

n

-RRB-

be

the

corresponding

eigenvectors

Matrix

H

Z

T

T

Z

is

also

a

symmetric

matrix

Let

k

denote

its

eigenvalues

According

to

Poincar

e

Separation

Theorem

we

know

the

relations

i

i

i

k

hold

Therefore

we

have

J

k

i

i

k

i

i

Similarly

we

have

J

k

i

i

k

i

i

In

both

cases

the

equations

hold

when

Z

-LRB-

v

v

k

-RRB-

R

where

R

is

an

arbitrary

k

k

orthonormal

matrix

Actually

the

solution

to

maximizing

J

is

just

the

well-known

theorem

of

Ky

Fan

-LRB-

the

Theorem

of

-LSB-

-RSB-

-RRB-

Note

that

the

optimal

Z

might

no

longer

conforms

to

the

definition

of

Z

ic

c

if

x

i

c

otherwise

but

it

is

still

a

orthonormal

matrix

That

is

why

it

is

called

a

relaxed

optimal

solution

clustering
large scale
concept modelling
support vector machines
clusters
trecvid
shrinking techniques
imbalance
imbalanced data

large scale data
meta-algorithm
kernel k-means
support vector machines (svms)
support cluster machines (scms)