Lossless

Online

Bayesian

Bagging

Bagging

frequently

improves

the

predictive

performance

of

a

model

An

online

version

has

recently

been

introduced

which

attempts

to

gain

the

benefits

of

an

online

algorithm

while

approximating

regular

bagging

However

regular

online

bagging

is

an

approximation

to

its

batch

counterpart

and

so

is

not

lossless

with

respect

to

the

bagging

operation

By

operating

under

the

Bayesian

paradigm

we

introduce

an

online

Bayesian

version

of

bagging

which

is

exactly

equivalent

to

the

batch

Bayesian

version

and

thus

when

combined

with

a

lossless

learning

algorithm

gives

a

completely

lossless

online

bagging

algorithm

We

also

note

that

the

Bayesian

formulation

resolves

a

theoretical

problem

with

bagging

produces

less

variability

in

its

estimates

and

can

improve

predictive

performance

for

smaller

data

sets

In

a

typical

prediction

problem

there

is

a

trade-off

between

bias

and

variance

in

that

after

a

certain

amount

of

fitting

any

increase

in

the

precision

of

the

fit

will

cause

an

increase

in

the

prediction

variance

on

future

observations

Similarly

any

reduction

in

the

prediction

variance

causes

an

increase

in

the

expected

bias

for

future

predictions

Breiman

-LRB-

-RRB-

introduced

bagging

as

a

method

of

reducing

the

prediction

variance

without

affecting

the

prediction

bias

As

a

result

predictive

performance

can

be

significantly

improved

Bagging

short

for

Bootstrap

AGGregatING

is

an

ensemble

learning

method

Instead

of

making

predictions

from

a

single

model

fit

on

the

observed

data

bootstrap

samples

are

taken

of

the

data

the

model

is

fit

on

each

sample

and

the

predictions

are

averaged

over

all

of

the

fitted

models

to

get

the

bagged

prediction

Breiman

-LRB-

-RRB-

explains

that

bagging

works

well

for

unstable

modeling

procedures

i.e.

those

for

which

the

conclusions

are

sensitive

to

small

changes

in

the

data

He

also

gives

a

theoretical

explanation

of

how

bagging

works

demonstrating

the

reduction

in

mean-squared

prediction

error

for

unstable

c

Herbert

K.

H.

Lee

and

Merlise

A.

Clyde

Lee

and

Clyde

procedures

Breiman

-LRB-

-RRB-

demonstrated

that

tree

models

among

others

are

empirically

unstable

Online

bagging

-LRB-

Oza

and

Russell

-RRB-

was

developed

to

implement

bagging

sequentially

as

the

observations

appear

rather

than

in

batch

once

all

of

the

observations

have

arrived

It

uses

an

asymptotic

approximation

to

mimic

the

results

of

regular

batch

bagging

and

as

such

it

is

not

a

lossless

algorithm

Online

algorithms

have

many

uses

in

modern

computing

By

updating

sequentially

the

update

for

a

new

observation

is

relatively

quick

compared

to

re-fitting

the

entire

database

making

real-time

calculations

more

feasible

Such

algorithms

are

also

advantageous

for

extremely

large

data

sets

where

reading

through

the

data

just

once

is

time-consuming

so

batch

algorithms

which

would

require

multiple

passes

through

the

data

would

be

infeasible

In

this

paper

we

consider

a

Bayesian

version

of

bagging

-LRB-

Clyde

and

Lee

-RRB-

based

on

the

Bayesian

bootstrap

-LRB-

Rubin

-RRB-

This

overcomes

a

technical

difficulty

with

the

usual

bootstrap

in

bagging

It

also

leads

to

a

theoretical

reduction

in

variance

over

the

bootstrap

for

certain

classes

of

estimators

and

a

significant

reduction

in

observed

variance

and

error

rates

for

smaller

data

sets

We

present

an

online

version

which

when

combined

with

a

lossless

online

model-fitting

algorithm

continues

to

be

lossless

with

respect

to

the

bagging

operation

in

contrast

to

ordinary

online

bagging

The

Bayesian

approach

requires

the

learning

base

algorithm

to

accept

weighted

samples

In

the

next

section

we

review

the

basics

of

the

bagging

algorithm

of

online

bagging

and

of

Bayesian

bagging

Next

we

introduce

our

online

Bayesian

bagging

algorithm

We

then

demonstrate

its

efficacy

with

classification

trees

on

a

variety

of

examples

In

ordinary

-LRB-

batch

-RRB-

bagging

bootstrap

re-sampling

is

used

to

reduce

the

variability

of

an

unstable

estimator

A

particular

model

or

algorithm

such

as

a

classification

tree

is

specified

for

learning

from

a

set

of

data

and

producing

predictions

For

a

particular

data

set

X

m

denote

the

vector

of

predictions

-LRB-

at

the

observed

sites

or

at

new

locations

-RRB-

by

G

-LRB-

X

m

-RRB-

Denote

the

observed

data

by

X

-LRB-

x

x

n

-RRB-

A

bootstrap

sample

of

the

data

is

a

sample

with

replacement

so

that

X

m

-LRB-

x

m

x

m

n

-RRB-

where

m

i

-LCB-

n

-RCB-

with

repetitions

allowed

X

m

can

also

be

thought

of

as

a

re-weighted

version

of

X

where

the

weights

-LRB-

m

-RRB-

i

are

drawn

from

the

set

-LCB-

n

n

-RCB-

i.e.

n

-LRB-

m

-RRB-

i

is

the

number

of

times

that

x

i

appears

in

the

mth

bootstrap

sample

We

denote

the

weighted

sample

as

-LRB-

X

-LRB-

m

-RRB-

-RRB-

For

each

bootstrap

sample

the

model

produces

predictions

G

-LRB-

X

m

-RRB-

G

-LRB-

X

m

-RRB-

G

-LRB-

X

m

-RRB-

P

where

P

is

the

number

of

prediction

sites

M

total

bootstrap

samples

are

used

The

bagged

predictor

for

the

jth

element

is

then

M

M

m

G

-LRB-

X

m

-RRB-

j

M

M

m

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

j

or

in

the

case

of

classification

the

jth

element

is

predicted

to

belong

to

the

most

frequently

predicted

category

by

G

-LRB-

X

-RRB-

j

G

-LRB-

X

M

-RRB-

j

A

version

of

pseudocode

for

implementing

bagging

is

For

m

-LCB-

M

-RCB-

Lossless

Online

Bayesian

Bagging

-LRB-

a

-RRB-

Draw

a

bootstrap

sample

X

m

from

X.

-LRB-

b

-RRB-

Find

predicted

values

G

-LRB-

X

m

-RRB-

The

bagging

predictor

is

M

M

m

G

-LRB-

X

m

-RRB-

Equivalently

the

bootstrap

sample

can

be

converted

to

a

weighted

sample

-LRB-

X

-LRB-

m

-RRB-

-RRB-

where

the

weights

-LRB-

m

-RRB-

i

are

found

by

taking

the

number

of

times

x

i

appears

in

the

bootstrap

sample

and

dividing

by

n.

Thus

the

weights

will

be

drawn

from

-LCB-

n

n

-RCB-

and

will

sum

to

The

bagging

predictor

using

the

weighted

formulation

is

M

M

m

G

-LRB-

X

m

-LRB-

m

-RRB-

-RRB-

for

regression

or

the

plurality

vote

for

classification

Online

Bagging

Online

bagging

-LRB-

Oza

and

Russell

-RRB-

was

recently

introduced

as

a

sequential

approximation

to

batch

bagging

In

batch

bagging

the

entire

data

set

is

collected

and

then

bootstrap

samples

are

taken

from

the

whole

database

An

online

algorithm

must

process

observations

as

they

arrive

and

thus

each

observation

must

be

resampled

a

random

number

of

times

when

it

arrives

The

algorithm

proposed

by

Oza

and

Russell

resamples

each

observation

according

to

a

Poisson

random

variable

with

mean

i.e.

P

-LRB-

K

m

k

-RRB-

exp

-LRB-

-RRB-

k

where

K

m

is

the

number

of

resamples

in

bootstrap

sample

m

K

m

-LCB-

-RCB-

Thus

as

each

observation

arrives

it

is

added

K

m

times

to

X

m

and

then

G

-LRB-

X

m

-RRB-

is

updated

and

this

is

done

for

m

-LCB-

M

-RCB-

Pseudocode

for

online

bagging

is

For

i

-LCB-

n

-RCB-

For

m

-LCB-

M

-RCB-

-LRB-

a

-RRB-

Draw

a

weight

K

m

from

a

Poisson

-LRB-

-RRB-

random

variable

and

add

K

m

copies

of

x

i

to

X

m

-LRB-

b

-RRB-

Find

predicted

values

G

-LRB-

X

m

-RRB-

The

current

bagging

predictor

is

M

M

m

G

-LRB-

X

m

-RRB-

Ideally

step

-LRB-

b

-RRB-

is

accomplished

with

a

lossless

online

update

that

incorporates

the

K

m

new

points

without

refitting

the

entire

model

We

note

that

n

may

not

be

known

ahead

of

time

but

the

bagging

predictor

is

a

valid

approximation

at

each

step

Online

bagging

is

not

guaranteed

to

produce

the

same

results

as

batch

bagging

In

particular

it

is

easy

to

see

that

after

n

points

have

been

observed

there

is

no

guarantee

that

X

m

will

contain

exactly

n

points

as

the

Poisson

weights

are

not

constrained

to

add

up

to

n

like

a

regular

bootstrap

sample

While

it

has

been

shown

-LRB-

Oza

and

Russell

-RRB-

that

these

samples

converge

asymptotically

to

the

appropriate

bootstrap

samples

there

may

be

some

discrepancy

in

practice

Thus

while

it

can

be

combined

with

a

lossless

online

learning

algorithm

-LRB-

such

as

for

a

classification

tree

-RRB-

the

bagging

part

of

the

online

ensemble

procedure

is

not

lossless

Lee

and

Clyde

Bayesian

Bagging

Ordinary

bagging

is

based

on

the

ordinary

bootstrap

which

can

be

thought

of

as

replacing

the

original

weights

of

n

on

each

point

with

weights

from

the

set

-LCB-

n

n

-RCB-

with

the

total

of

all

weights

summing

to

A

variation

is

to

replace

the

ordinary

bootstrap

with

the

Bayesian

bootstrap

-LRB-

Rubin

-RRB-

The

Bayesian

approach

treats

the

vector

of

weights

as

unknown

parameters

and

derives

a

posterior

distribution

for

and

hence

G

-LRB-

X

-RRB-

The

non-informative

prior

n

i

i

when

combined

with

the

multinomial

likelihood

leads

to

a

Dirichlet

n

-LRB-

-RRB-

distribution

for

the

posterior

distribution

of

The

full

posterior

distribution

of

G

-LRB-

X

-RRB-

can

be

estimated

by

Monte

Carlo

methods

generate

-LRB-

m

-RRB-

from

a

Dirichlet

n

-LRB-

-RRB-

distribution

and

then

calculate

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

for

each

sample

The

average

of

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

over

the

M

samples

corresponds

to

the

Monte

Carlo

estimate

of

the

posterior

mean

of

G

-LRB-

X

-RRB-

and

can

be

viewed

as

a

Bayesian

analog

of

bagging

-LRB-

Clyde

and

Lee

-RRB-

In

practice

we

may

only

be

interested

in

a

point

estimate

rather

than

the

full

posterior

distribution

In

this

case

the

Bayesian

bootstrap

can

be

seen

as

a

continuous

version

of

the

regular

bootstrap

Thus

Bayesian

bagging

can

be

achieved

by

generating

M

Bayesian

bootstrap

samples

and

taking

the

average

or

majority

vote

of

the

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

This

is

identical

to

regular

bagging

except

that

the

weights

are

continuous-valued

on

-LRB-

-RRB-

instead

of

being

restricted

to

the

discrete

set

-LCB-

n

n

-RCB-

In

both

cases

the

weights

must

sum

to

In

both

cases

the

expected

value

of

a

particular

weight

is

n

for

all

weights

and

the

expected

correlation

between

weights

is

the

same

-LRB-

Rubin

-RRB-

Thus

Bayesian

bagging

will

generally

have

the

same

expected

point

estimates

as

ordinary

bagging

The

variability

of

the

estimate

is

slightly

smaller

under

Bayesian

bagging

as

the

variability

of

the

weights

is

n

n

times

that

of

ordinary

bagging

As

the

sample

size

grows

large

this

factor

becomes

arbitrarily

close

to

one

but

we

do

note

that

it

is

strictly

less

than

one

so

the

Bayesian

approach

does

give

a

further

reduction

in

variance

compared

to

the

standard

approach

In

practice

for

smaller

data

sets

we

often

find

a

significant

reduction

in

variance

possibly

because

the

use

of

continuous-valued

weights

leads

to

fewer

extreme

cases

than

discrete-valued

weights

Pseudocode

for

Bayesian

bagging

is

For

m

-LCB-

M

-RCB-

-LRB-

a

-RRB-

Draw

random

weights

-LRB-

m

-RRB-

from

a

Dirichlet

n

-LRB-

-RRB-

to

produce

the

Bayesian

bootstrap

sample

-LRB-

X

-LRB-

m

-RRB-

-RRB-

-LRB-

b

-RRB-

Find

predicted

values

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

The

bagging

predictor

is

M

M

m

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

Use

of

the

Bayesian

bootstrap

does

have

a

major

theoretical

advantage

in

that

for

some

problems

bagging

with

the

regular

bootstrap

is

actually

estimating

an

undefined

quantity

To

take

a

simple

example

suppose

one

is

bagging

the

fitted

predictions

for

a

point

y

from

a

least-squares

regression

problem

Technically

the

full

bagging

estimate

is

M

m

y

m

where

m

ranges

over

all

possible

bootstrap

samples

M

is

the

total

number

of

possible

bootstrap

samples

and

y

m

is

the

predicted

value

from

the

model

fit

using

the

mth

bootstrap

sample

The

issue

is

that

one

of

the

possible

bootstrap

samples

contains

the

Lossless

Online

Bayesian

Bagging

first

data

point

replicated

n

times

and

no

other

data

points

For

this

bootstrap

sample

the

regression

model

is

undefined

-LRB-

since

at

least

two

different

points

are

required

-RRB-

and

so

y

and

thus

the

bagging

estimator

are

undefined

In

practice

only

a

small

sample

of

the

possible

bootstrap

samples

is

used

so

the

probability

of

drawing

a

bootstrap

sample

with

an

undefined

prediction

is

very

small

Yet

it

is

disturbing

that

in

some

problems

the

bagging

estimator

is

technically

not

well-defined

In

contrast

the

use

of

the

Bayesian

bootstrap

completely

avoids

this

problem

Since

the

weights

are

continuous-valued

the

probability

that

any

weight

is

exactly

equal

to

zero

is

zero

Thus

with

probability

one

all

weights

are

strictly

positive

and

the

Bayesian

bagging

estimator

will

be

well-defined

-LRB-

assuming

the

ordinary

estimator

on

the

original

data

is

well-defined

-RRB-

We

note

that

the

Bayesian

approach

will

only

work

with

models

that

have

learning

algorithms

that

handle

weighted

samples

Most

standard

models

either

have

readily

available

such

algorithms

or

their

algorithms

are

easily

modified

to

accept

weights

so

this

restriction

is

not

much

of

an

issue

in

practice

Regular

online

bagging

can

not

be

exactly

equivalent

to

the

batch

version

because

the

Poisson

counts

can

not

be

guaranteed

to

sum

to

the

number

of

actual

observations

Gamma

random

variables

can

be

thought

of

as

continuous

analogs

of

Poisson

counts

which

motivates

our

derivation

of

Bayesian

online

bagging

The

key

is

to

recall

a

fact

from

basic

probability

a

set

of

independent

gamma

random

variables

divided

by

its

sum

has

a

Dirichlet

distribution

i.e.

If

w

i

-LRB-

i

-RRB-

then

w

w

i

w

w

i

w

k

w

i

Dirichlet

n

-LRB-

k

-RRB-

-LRB-

See

for

example

Hogg

and

Craig

pp.

-RRB-

This

relationship

is

a

common

method

for

generating

random

draws

from

a

Dirichlet

distribution

and

so

is

also

used

in

the

implementation

of

batch

Bayesian

bagging

in

practice

Thus

in

the

online

version

of

Bayesian

bagging

as

each

observation

arrives

it

has

a

realization

of

a

Gamma

-LRB-

-RRB-

random

variable

associated

with

it

for

each

bootstrap

sample

and

the

model

is

updated

after

each

new

weighted

observation

If

the

implementation

of

the

model

requires

weights

that

sum

to

one

then

within

each

-LRB-

Bayesian

-RRB-

bootstrap

sample

all

weights

can

be

re-normalized

with

the

new

sum

of

gammas

before

the

model

is

updated

At

any

point

in

time

the

current

predictions

are

those

aggregated

across

all

bootstrap

samples

just

as

with

batch

bagging

If

the

model

is

fit

with

an

ordinary

lossless

online

algorithm

as

exists

for

classification

trees

-LRB-

Utgoff

et

al.

-RRB-

then

the

entire

online

Bayesian

bagging

procedure

is

completely

lossless

relative

to

batch

Bayesian

bagging

Furthermore

since

batch

Bayesian

bagging

gives

the

same

mean

results

as

ordinary

batch

bagging

online

Bayesian

bagging

also

has

the

same

expected

results

as

ordinary

batch

bagging

Pseudocode

for

online

Bayesian

bagging

is

For

i

-LCB-

n

-RCB-

For

m

-LCB-

M

-RCB-

Lee

and

Clyde

-LRB-

a

-RRB-

Draw

a

weight

-LRB-

m

-RRB-

i

from

a

Gamma

-LRB-

-RRB-

random

variable

associate

weight

with

x

i

and

add

x

i

to

X.

-LRB-

b

-RRB-

Find

predicted

values

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

-LRB-

renormalizing

weights

if

necessary

-RRB-

The

current

bagging

predictor

is

M

M

m

G

-LRB-

X

-LRB-

m

-RRB-

-RRB-

In

step

-LRB-

b

-RRB-

the

weights

may

need

to

be

renormalized

-LRB-

by

dividing

by

the

sum

of

all

current

weights

-RRB-

if

the

implementation

requires

weights

that

sum

to

one

We

note

that

for

many

models

such

as

classification

trees

this

renormalization

is

not

a

major

issue

for

a

tree

each

split

only

depends

on

the

relative

weights

of

the

observations

at

that

node

so

nodes

not

involving

the

new

observation

will

have

the

same

ratio

of

weights

before

and

after

renormalization

and

the

rest

of

the

tree

structure

will

be

unaffected

in

practice

in

most

implementations

of

trees

-LRB-

including

that

used

in

this

paper

-RRB-

renormalization

is

not

necessary

We

discuss

the

possibility

of

renormalization

in

order

to

be

consistent

with

the

original

presentation

of

the

bootstrap

and

Bayesian

bootstrap

and

we

note

that

ordinary

online

bagging

implicitly

deals

with

this

issue

equivalently

The

computational

requirements

of

Bayesian

versus

ordinary

online

bagging

are

comparable

The

procedures

are

quite

similar

with

the

main

difference

being

that

the

fitting

algorithm

must

handle

non-integer

weights

for

the

Bayesian

version

For

models

such

as

trees

there

is

no

significant

additional

computational

burden

for

using

non-integer

weights

We

demonstrate

the

effectiveness

of

online

Bayesian

bagging

using

classification

trees

Our

implementation

uses

the

lossless

online

tree

learning

algorithms

-LRB-

ITI

-RRB-

of

Utgoff

et

al.

-LRB-

-RRB-

-LRB-

available

at

-RRB-

We

compared

Bayesian

bagging

to

a

single

tree

ordinary

batch

bagging

and

ordinary

online

bagging

all

three

of

which

were

done

using

the

minimum

description

length

criterion

-LRB-

MDL

-RRB-

as

implemented

in

the

ITI

code

to

determine

the

optimal

size

for

each

tree

To

implement

Bayesian

bagging

the

code

was

modified

to

account

for

weighted

observations

We

use

a

generalized

MDL

to

determine

the

optimal

tree

size

at

each

stage

replacing

all

counts

of

observations

with

the

sum

of

the

weights

of

the

observations

at

that

node

or

leaf

with

the

same

response

category

Replacing

the

total

count

directly

with

the

sum

of

the

weights

is

justified

by

looking

at

the

multinomial

likelihood

when

written

as

an

exponential

family

in

canonical

form

the

weights

enter

through

the

dispersion

parameter

and

it

is

easily

seen

that

the

unweighted

counts

are

replaced

by

the

sums

of

the

weights

of

the

observations

that

go

into

each

count

To

be

more

specific

a

decision

tree

typically

operates

with

a

multinomial

likelihood

leaves

j

classes

k

p

n

jk

jk

where

p

jk

is

the

true

probability

that

an

observation

in

leaf

j

will

be

in

class

k

and

n

jk

is

the

count

of

data

points

in

leaf

j

in

class

k

This

is

easily

re-written

as

the

product

over

all

observations

n

i

p

i

where

if

observation

i

is

in

leaf

j

and

a

member

of

class

k

then

p

i

p

jk

For

simplicity

we

consider

the

case

k

as

the

generalization

to

larger

k

is

straightforward

Now

consider

a

single

point

y

which

takes

values

or

depending

on

which

class

is

it

a

member

of

Transforming

to

the

canonical

parameterization

let

p

Lossless

Online

Bayesian

Bagging

where

p

is

the

true

probability

that

y

Writing

the

likelihood

in

exponential

family

form

gives

exp

y

log

exp

-LCB-

-RCB-

a

where

a

is

the

dispersion

parameter

which

would

be

equal

to

for

a

standard

data

set

but

would

be

the

reciprocal

of

the

weight

for

that

observation

in

a

weighted

data

set

Thus

the

likelihood

for

an

observation

y

with

weight

w

is

exp

y

log

exp

-LCB-

-RCB-

-LRB-

-RRB-

p

wy

-LRB-

p

-RRB-

w

-LRB-

-RRB-

and

so

returning

to

the

full

multinomial

the

original

counts

are

simply

replaced

by

the

weighted

counts

As

MDL

is

a

penalized

likelihood

criterion

we

thus

use

the

weighted

likelihood

and

replace

each

count

with

a

sum

of

weights

We

note

that

for

ordinary

online

bagging

using

a

single

Poisson

weight

K

with

our

generalized

MDL

is

exactly

equivalent

to

including

K

copies

of

the

data

point

in

the

data

set

and

using

regular

MDL

Table

shows

the

data

sets

we

used

for

classification

problems

the

number

of

classes

in

each

data

set

and

the

sizes

of

their

respective

training

and

test

partitions

Table

displays

the

results

of

our

comparison

study

All

of

the

data

sets

except

the

final

one

are

available

online

at

the

UCI

Machine

Learning

Repository

The

last

data

set

is

described

in

Lee

-LRB-

-RRB-

We

compare

the

results

of

training

a

single

classification

tree

ordinary

batch

bagging

online

bagging

and

Bayesian

online

bagging

-LRB-

or

equivalently

Bayesian

batch

-RRB-

For

each

of

the

bagging

techniques

bootstrap

samples

were

used

For

each

data

set

we

repeated

times

the

following

procedure

randomly

choose

a

training/test

partition

fit

a

single

tree

a

batch

bagged

tree

an

online

bagged

tree

and

a

Bayesian

bagged

tree

compute

the

misclassification

error

rate

for

each

fit

Table

reports

the

average

error

rate

for

each

method

on

each

data

set

as

well

as

the

estimated

standard

error

of

this

error

rate

Size

of

Size

of

Number

of

Training

Test

Data

Set

Classes

Data

Set

Data

Set

Breast

cancer

-LRB-

WI

-RRB-

Contraceptive

Credit

-LRB-

German

-RRB-

Credit

-LRB-

Japanese

-RRB-

Dermatology

Glass

House

votes

Ionosphere

Iris

Liver

Pima

diabetes

SPECT

Wine

Mushrooms

Spam

Credit

-LRB-

American

-RRB-

Table

Sizes

of

the

example

data

sets

Lee

and

Clyde

Bayesian

Single

Batch

Online

Online/Batch

Data

Set

Tree

Bagging

Bagging

Bagging

Breast

cancer

-LRB-

WI

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Contraceptive

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Credit

-LRB-

German

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Credit

-LRB-

Japanese

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Dermatology

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Glass

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

House

votes

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Ionosphere

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Iris

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Liver

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Pima

diabetes

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

SPECT

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Wine

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Mushrooms

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Spam

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Credit

-LRB-

American

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

Comparison

of

average

classification

error

rates

-LRB-

with

standard

error

-RRB-

We

note

that

in

all

cases

both

online

bagging

techniques

produce

results

similar

to

ordinary

batch

bagging

and

all

bagging

methods

significantly

improve

upon

the

use

of

a

single

tree

However

for

smaller

data

sets

-LRB-

all

but

the

last

three

-RRB-

online/batch

Bayesian

bagging

typically

both

improves

prediction

performance

and

decreases

prediction

variability

Bagging

is

a

useful

ensemble

learning

tool

particularly

when

models

sensitive

to

small

changes

in

the

data

are

used

It

is

sometimes

desirable

to

be

able

to

use

the

data

in

an

online

fashion

By

operating

in

the

Bayesian

paradigm

we

can

introduce

an

online

algorithm

that

will

exactly

match

its

batch

Bayesian

counterpart

Unlike

previous

versions

of

online

bagging

the

Bayesian

approach

produces

a

completely

lossless

bagging

algorithm

It

can

also

lead

to

increased

accuracy

and

decreased

prediction

variance

for

smaller

data

sets

This

research

was

partially

supported

by

NSF

grants

DMS

and

The

authors

would

like

to

thank

two

anonymous

referees

for

their

helpful

suggestions

Lossless

Online

Bayesian

Bagging

L.

Breiman

Heuristics

of

instability

in

model

selection

Technical

report

University

of

California

at

Berkeley

L.

Breiman

Bagging

predictors

Machine

Learning

-LRB-

-RRB-

M.

A.

Clyde

and

H.

K.

H.

Lee

Bagging

and

the

Bayesian

bootstrap

In

T.

Richardson

and

T.

Jaakkola

editors

Artificial

Intelligence

and

Statistics

pages

R.

V.

Hogg

and

A.

T.

Craig

Introduction

to

Mathematical

Statistics

Prentice-Hall

Upper

Saddle

River

NJ

edition

H.

K.

H.

Lee

Model

selection

for

neural

network

classification

Journal

of

Classification

N.

C.

Oza

and

S.

Russell

Online

bagging

and

boosting

In

T.

Richardson

and

T.

Jaakkola

editors

Artificial

Intelligence

and

Statistics

pages

D.

B.

Rubin

The

Bayesian

bootstrap

Annals

of

Statistics

P.

E.

Utgoff

N.

C.

Berkman

and

J.

A.

Clouse

Decision

tree

induction

based

on

efficient

tree

restructuring

Machine

Learning

O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I I O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O B O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O B O O O O O O O B O O O O O O O O B O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O B O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O B O O O O O O O O B O O O B O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O B O O O O B O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

lossless learning algorithm
bayesian bootstrap
bagging
classification
dirichlet distribution
bootstrap

mean-squared prediction error
classification tree
online bagging
bayesian bagging


A Geometric Constraint Library for 3D Graphical Applications Recent computer technologies have enabled fast high-quality 3D graphics on personal computers , and also have made the development of 3D graphical applications easier . However , most of such technologies do not sufficiently support layout and behavior aspects of 3D graphics . Geometric constraints are , in general , a powerful tool for specifying layouts and behaviors of graphical objects , and have been applied to 2D graphical user interfaces and specialized 3D graphics packages . In this paper , we present Chorus3D , a geometric constraint library for 3D graphical applications . It enables programmers to use geometric constraints for various purposes such as geometric layout , constrained dragging , and inverse kinematics . Its novel feature is to handle scene graphs by processing coordinate transformations in geometric constraint satisfaction . We demonstrate the usefulness of Chorus3D by presenting sample constraint-based 3D graphical applications . Recent advances in commodity hardware have enabled fast high-quality 3D graphics on personal computers . Also , software technologies such as VRML and Java 3D have made the development of 3D graphical applications easier . However , most of such technologies mainly focus on rendering aspects of 3D graphics , and do not sufficiently support layout and behavior aspects . Constraints are , in general , a powerful tool for specifying layouts and behaviors of graphical objects . It is widely recognized that constraints facilitate describing geometric layouts and behaviors of diagrams in 2D graphical user interfaces such as drawing editors , and therefore constraint solvers for this purpose have been extensively studied -LSB- 3 , 7 , Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to distribute to lists , requires prior specific permission and/or fee . Int . Symp . on Smart Graphics , June 11-13 , 2002 , Hawthorne , NY , USA . Copyright 2002 ACM 1-58113-555-6 / 02/0600 ... $ 5.00 8 , 9 , 11 , 12 , 13 , 17 , 18 -RSB- . Also , many specialized 3D graphics packages enable the specification of object layouts and behaviors by using constraints or similar functions . It is natural to consider that various 3D graphical applications can also be enhanced by incorporating constraints . It might seem sufficient for this purpose to modify existing 2D geometric constraint solvers to support 3D geometry . It is , however , insufficient in reality because of the essential difference between the ways of specifying 2D and 3D graphics ; typical 2D graphics handles only simple coordinate systems , whereas most 3D graphics requires multiple coordinate systems with complex relations such as rotations to treat scene graphs . It means that we need to additionally support coordinate transformations in 3D geometric constraint solvers . In this paper , we present Chorus3D , a geometric constraint library for 3D graphical applications . The novel feature of Chorus3D is to handle scene graphs by processing coordinate transformations in geometric constraint satisfaction . We have realized Chorus3D by adding this feature to our previous 2D geometric constraint library Chorus -LSB- 13 -RSB- . Another important point of Chorus3D is that it inherits from Chorus the capability to handle `` soft '' constraints with hierarchical strengths or preferences -LRB- i.e. , constraint hierarchies -LSB- 7 -RSB- -RRB- , which are useful for specifying default layouts and behaviors of graphical objects . It determines solutions so that they satisfy as many strong constraints as possible , leaving weaker inconsistent constraints unsatisfied . Chorus3D also inherits from Chorus a module mechanism which allows user-defined kinds of geometric constraints . This feature enables programmers to use geometric constraints for various purposes including the following : Geometric layout : A typical use of Chorus3D is to lay out graphical objects . For example , it allows putting objects parallel or perpendicular to others without requiring predetermined positioning parameters . Also , it provides constraint-based general graph layout based on the spring model -LSB- 14 -RSB- . Constrained dragging : Chorus3D enables dragging objects with positioning constraints . For example , it can constrain a dragged object to be on the surface of a sphere . Constrained dragging is important for 3D graphics because it provides a sophisticated way to ac-94 commodate ordinary mouse dragging to 3D spaces . Inverse kinematics : Chorus3D is applicable to inverse kinematics , which is a problem of finding desired configurations of `` articulated '' objects -LSB- 1 , 20 -RSB- . It allows the specification of articulated objects by using coordinate transformations , and can automatically calculate the parameters of the transformations that satisfy constraints . This method is also applicable to camera control by aiming at a possibly moving target object . In this paper , we demonstrate the usefulness of Chorus3D by presenting sample constraint-based 3D graphical applications . This paper is organized as follows : We first present our approach to the use of constraints for 3D graphics . Second , we describe our basic framework of constraints . Next , we present a method for processing coordinate transformations in our framework . We then provide the implementation of Chorus3D , and demonstrate examples of using constraints in 3D graphics . After giving related work and discussion , we mention the conclusions and future work of this research . In this research , we integrate geometric constraints with 3D graphics . Basically , we realize this by extending our previous 2D geometric constraint solver Chorus -LSB- 13 -RSB- to support 3D geometry . However , as already mentioned , it is not a straightforward task because 3D graphics typically requires handling scene graphs with hierarchical structures of coordinate systems , which is not covered by the 2D version of the Chorus constraint solver . To support hierarchies of coordinate systems , we introduce the following new model of constraints : Point variables : Each point variable -LRB- which consists of three real-valued constrainable variables -RRB- is associated with one coordinate system , and its value is expressed as local coordinates . Geometric constraints : Geometric constraints on point variables are evaluated by using the world coordinates of the point variables -LRB- they can also refer to 1D variables for , e.g. , distances and angles by using their values directly -RRB- . A single constraint can refer to point variables belonging to different coordinate systems . Coordinate transformations : Parameters of coordinate transformations are provided as constrainable variables , and the solver is allowed to change the parameters of transformations to appropriately satisfy given constraints . With this model , we can gain the benefit of the easy maintenance of geometric relations by using constraints , as well as the convenience of modeling geometric objects by employing scene graphs . In our actual implementation , we provide the following three elemental kinds of coordinate transformations : Translation : A translation transformation is characterized with three variables t x , t y , and t z , and specifies the translation of vector -LRB- t x , t y , t z -RRB- . Rotation : A rotation transformation is parameterized with four variables r x , r y , r z , and r w , and specifies the rotation of angle r w about the axis -LRB- r x , r y , r z -RRB- . Scale : A scale transformation is represented with three variables s x , s y , and s z , and specifies the axis-wise scale -LRB- s x , s y , s z -RRB- about the origin . We can express many practically useful transformations by using such elemental ones . In fact , any transformations represented with Transform nodes in VRML can be realized by combining these kinds of transformations -LSB- 4 -RSB- . In this section , we briefly describe our framework for handling constraints . We base it on the framework for the 2D version of the Chorus constraint solver . See -LSB- 13 -RSB- for further detail . 3.1 Problem Formulation We first present the mathematical formulation for modeling constraints and constraint systems . In the following , we write x to represent a variable vector -LRB- x 1 , x 2 , ... , x n -RRB- of n variables , and also v to indicate a variable value vector -LRB- v 1 , v 2 , ... , v n -RRB- of n real numbers -LRB- v i expresses the value of x i -RRB- . To support various geometric constraints in a uniform manner , we adopt error functions as a means of expressing constraints . An error function e -LRB- x -RRB- is typically associated with a single arithmetic constraint , and is defined as a function from variable value vectors to errors expressed as non-negative real numbers ; that is , e -LRB- v -RRB- gives the error of the associated constraint for v . An error function returns a zero if and only if the constraint is exactly satisfied . For example , e -LRB- x -RRB- = -LRB- x i - x j -RRB- 2 can be used for the constraint x i = x j . We assume that , for each e -LRB- x -RRB- , its gradient is known : e -LRB- x -RRB- = e -LRB- x -RRB- x 1 , e -LRB- x -RRB- x 2 , ... , e -LRB- x -RRB- x n . In the same way as constraint hierarchies -LSB- 7 -RSB- , constraint systems in our framework can be divided into levels consisting of constraints with equal strengths . Constraints with the strongest preference are said to be required -LRB- or hard -RRB- , and are guaranteed to be always satisfied -LRB- if it is impossible , there will be no solution -RRB- . By contrast , constraints with weaker preferences are said to be preferential -LRB- or soft -RRB- , and may be relaxed if they conflict with stronger constraints . Solutions to constraint systems are defined as follows : let e i , j -LRB- x -RRB- be the error function of the j-th constraint -LRB- 1 j m i -RRB- at strength level i -LRB- 0 i l -RRB- ; then solutions v are determined with the optimization problem minimize v E -LRB- v -RRB- subject to e 0 , j -LRB- v -RRB- = 0 -LRB- 1 j m 0 -RRB- 95 where E is an objective function defined as E -LRB- x -RRB- = l i = 1 m i j = 1 w i e i , j -LRB- x -RRB- in which w i indicates the weight associated with strength i , and the relation w 1 w 2 w l holds . In this formulation , level 0 corresponds to required constraints , and the others to preferential ones . Intuitively , more weighted -LRB- or stronger -RRB- preferential constraints should be more satisfied . Our framework simulates constraint hierarchies . Particularly , if the squares of constraint violations are used to compute error functions , a system in our framework will obtain approximate solutions to the similar hierarchy solved with the criterion least-squares-better -LSB- 3 , 17 -RSB- . The largest difference is that a system in our framework slightly considers a weak constraint inconsistent with a stronger satisfiable one in computing its solutions , while the similar hierarchy would discard such a weak one . Our actual implementation of the Chorus3D constraint solver provides four external strengths required , strong , medium , and weak as well as two internal strengths very strong -LRB- used to approximately handle required nonlinear or inequality constraints -RRB- and very weak -LRB- exploited to make new solutions as close to previous ones as possible -RRB- . It typically assigns weights 32 4 , 32 3 , 32 2 , 32 1 , and 1 to strengths very strong , strong , medium , weak , and very weak respectively . These weights were determined according to the precision of the actual numerical algorithm -LRB- described in the next subsection -RRB- . To know how much these weights affect solutions , suppose a system of strong constraint x = 0 and medium one x = 100 . Then the unique solution will be obtained as x = 3.0303 -LRB- = 100/33 -RRB- . Thus the difference of strengths is obvious . According to our actual experience , this precision allows us to discriminate constraint strengths in most graphical applications . 3.2 Algorithm To actually find solutions to constraint systems presented above , we need to solve their corresponding optimization problems . For this purpose , we designed a constraint satisfaction algorithm by combining a numerical optimization technique with a genetic algorithm . It uses numerical optimization to find local solutions , while it adopts a genetic algorithm to search for global solutions . For numerical optimization , we mainly use the quasi-Newton method based on Broyden-Fletcher-Goldfarb-Sahnno updating formula -LSB- 2 , 6 -RSB- , which is a fast iterative technique that exhibits superlinear convergence . Since it excludes fruitless searches by utilizing its history , it is usually faster than straightforward Newton 's method . We introduced a genetic algorithm to alleviate the problem that some kinds of geometric constraints suffer from local optimal but global non-optimal solutions -LSB- 11 , 16 -RSB- . Generally , a genetic algorithm is a stochastic search method that repeatedly transforms a population of potential solutions into another next-generation population -LSB- 10 , 15 -RSB- . We typically necessitate it only for computing initial solutions ; in other words , we can usually re-solve modified constraint systems without the genetic algorithm , only by applying numerical optimization to previous solutions . In this section , we propose a method for integrating coordinate transformations with our constraint framework . As already mentioned , we use world coordinates of points to evaluate 3D geometric constraints . A naive method for this is to duplicate point variables in all ancestor coordinate systems , and then to impose required constraints that represent coordinate transformations between the point variables . However , this method requires an optimization routine supporting required nonlinear constraints , which limits the availability of actual techniques -LRB- in fact , we can not use the quasi-Newton method for this purpose -RRB- . Also , this method tends to yield many variables and constraints , and therefore requires an extra amount of memory . Below we propose a more widely applicable method for handling coordinate transformations . Its characteristic is to hide transformations from optimization routines , which is realized by embedding transformations in error functions . 4.1 Model To begin with , we introduce another variable vector x = -LRB- x 1 , x 2 , ... , x n -RRB- , which is created by replacing variables for local coordinates of 3D points in x with the corresponding ones for world coordinates -LRB- 1D variables remain the same -RRB- . We can mathematically model this process as follows : Consider the sequence of the s transformations y 0 -LRB- = x -RRB- t 0 - y 1 t 1 - t s-2 - y s-1 t s-1 - y s -LRB- = x -RRB- where y 0 and y s are equal to x and x respectively , each y k -LRB- 1 k s - 1 -RRB- is an `` intermediate '' vector , and each t k -LRB- 0 k s - 1 -RRB- is a function that transforms y k into y k +1 . Intuitively , t k corresponds to a coordinate transformation , and transforms related point variables from its source coordinate system into its destination system . It should be noted that , although transformations are , in general , hierarchical -LRB- or tree-structured -RRB- , we can always find such a linear sequence by `` serializing '' them in an appropriate order . By using such transformations , we can compute x as follows : x = t s-1 -LRB- t s-2 -LRB- -LRB- t 1 -LRB- t 0 -LRB- x -RRB- -RRB- -RRB- -RRB- -RRB- t -LRB- x -RRB- where t is defined as the composition of all the elemental transformations . In the following description , we write y k , i to denote the i-th element of y k , and also t k , i to represent the i-th element of t k ; that is , y k +1 = -LRB- y k +1,1 , y k +1,2 , ... , y k +1 , n -RRB- = -LRB- t k ,1 -LRB- y k -RRB- , t k ,2 -LRB- y k -RRB- , ... , t k , n -LRB- y k -RRB- -RRB- = t k -LRB- y k -RRB- . 4.2 Method Geometric constraints are evaluated by using world coordinates of points , which means that their error functions are 96 defined as e -LRB- x -RRB- . Using the composed transformations , we can evaluate them as e -LRB- x -RRB- = e -LRB- t -LRB- x -RRB- -RRB- . Importantly , we can efficiently realize this computation by applying only necessary transformations to actually used variables . We also need to compute the gradient of e -LRB- t -LRB- x -RRB- -RRB- , i.e. , e -LRB- t -LRB- x -RRB- -RRB- = e -LRB- t -LRB- x -RRB- -RRB- x 1 , e -LRB- t -LRB- x -RRB- -RRB- x 2 , ... , e -LRB- t -LRB- x -RRB- -RRB- x n . Basically , we can decompose each partial derivative e -LRB- t -LRB- x -RRB- -RRB- / x i into primitive expressions by repeatedly using the chain rule . However , we should avoid the simple application of the chain rule since it would result in a large number of expressions . Instead , we perform a controlled way of decomposing such partial derivatives ; it appropriately arranges the chain rule to restrict the computation to only necessary components . First , we decompose e -LRB- t -LRB- x -RRB- -RRB- / x i as follows : e -LRB- t -LRB- x -RRB- -RRB- x i = j e -LRB- x -RRB- x j t s-1 , j -LRB- y s-1 -RRB- x i = j e -LRB- x -RRB- x j j s-1 t s-1 , j -LRB- y s-1 -RRB- y s-1 , j s-1 t s-2 , j s-1 -LRB- y s-2 -RRB- x i = j s-1 j e -LRB- x -RRB- x j t s-1 , j -LRB- y s-1 -RRB- y s-1 , j s-1 t s-2 , j s-1 -LRB- y s-2 -RRB- x i = j s-1 e -LRB- x -RRB- y s-1 , j s-1 t s-2 , j s-1 -LRB- y s-2 -RRB- x i . Note that each e -LRB- x -RRB- / x j is given by the definition of the geometric constraint , and also that each t s-1 , j -LRB- y s-1 -RRB- / y s-1 , j s-1 is a partial derivative in the gradient of a single coordinate transformation t s-1 . Thus we can obtain each e -LRB- x -RRB- / y s-1 , j s-1 . Also , by repeating this process , we can compute , for each k , e -LRB- t -LRB- x -RRB- -RRB- x i = j k e -LRB- x -RRB- y k , j k t k-1 , j k -LRB- y k-1 -RRB- x i and finally achieve e -LRB- t -LRB- x -RRB- -RRB- x i = j 1 e -LRB- x -RRB- y 1 , j 1 t 0 , j 1 -LRB- x -RRB- x i where each t 0 , j 1 -LRB- x -RRB- / x i is a component of the gradient of t 0 . Therefore , e -LRB- t -LRB- x -RRB- -RRB- / x i is now determined . Furthermore , we can considerably reduce the number of the computations of e -LRB- x -RRB- / y k , j k in practice . We can make the following observations about the above computation : For each variable x j , e -LRB- x -RRB- / x j can be non-zero only if x j is actually needed to evaluate the designated constraint . If x i is originated in the coordinate system associated with t k -LRB- that is , x i is either a local coordinate or a parameter of the coordinate transformation -RRB- , we have y k , i = x i , which means that we have t k , j -LRB- y k -RRB- / x i . Therefore , we can compute e -LRB- x -RRB- / x i immediately . These observations reveal that we need to transfer a partial derivative e -LRB- x -RRB- / y k , j to the next step only when x j represents a really necessary coordinate that has not reached its local coordinate system . Also , since we can handle each necessary point independently , we can implement this process with a linear recursive function that hands over only three derivatives e -LRB- x -RRB- / y k , j at each recursive call . We implemented the proposed method by developing a constraint solver called Chorus3D , which is a 3D extension to our previous 2D geometric constraint solver Chorus -LSB- 13 -RSB- . We constructed Chorus3D as a C++ class library , and also developed a native method interface to make it available to Java programs . Chorus3D allows programmers to add a new kind of arithmetic constraints -LRB- e.g. , Euclidean geometric constraints -RRB- by constructing a new constraint class with a method that evaluates their error functions . Also , programmers can introduce a new kind of non-arithmetic -LRB- or pseudo -RRB- constraints -LRB- for , e.g. , general graph layout -RRB- by developing a new evalua-tion module which computes an `` aggregate '' error function for a given set of constraints . Chorus3D currently provides linear equality , linear inequality , edit -LRB- update a variable value -RRB- , stay -LRB- fix a variable value -RRB- , Euclidean geometric constraints -LRB- for , e.g. , parallelism , per-pendicularity , and distance equality -RRB- , and graph layout constraints based on the spring model -LSB- 14 -RSB- . Linear equality / inequality constraints can refer to only 1D variables -LRB- including elements of 3D point variables -RRB- , while edit and stay constraints can be associated with 1D and 3D point variables . Euclidean geometric constraints typically refer to point variables although they sometimes require 1D variables for angles and distances . Each graph layout constraint represents a graph edge , and refers to two point variables as its associated graph nodes . As stated earlier , constraints on such point variables are evaluated by using world coordinates of the points . Also , a single constraint can refer to point variables belonging to different coordinate systems . The application programming interface of Chorus3D is a natural extension to that of Chorus , which provides a certain compatibility with a recent linear solver called Cassowary -LSB- 3 -RSB- ; in a similar way to Cassowary and Chorus , Chorus3D allows programmers to process constraint systems by creating variables and constraints as objects , and by adding / removing constraint objects to/from the solver object . In addition , Chorus3D handles coordinate transformations as objects , and presents an interface for arranging them hier-archically In this section , we present three examples to demonstrate how to incorporate geometric constraints into 3D graphics by using the Chorus3D constraint solver . All the examples are implemented in Java by using Java 3D as a graphics 97 Figure 1 : A 3D geometric layout of a general graph structure . programming interface as well as the native method interface with Chorus3D . We also provide computation times taken for constraint satisfaction in these examples . 6.1 Graph Layout The first example is an application which lays out a set of points with a general graph structure in a 3D space as shown in Figure 1 . This application also allows a user to drag graph nodes with a mouse . 1 The used graph layout technique is based on a 3D extension to the spring model -LSB- 14 -RSB- . This kind of 3D graph layout is practically useful to information visualization , and has actually been adopted in a certain system -LSB- 19 -RSB- . The constraint system of this graph layout consists of 26 point variables -LRB- i.e. , 78 real-valued variables -RRB- , 31 graph layout constraints , and three linear equality constraints for fixing one of the point variables at the origin . When executed on an 866 MHz Pentium III processor running Linux 2.2.16 , Chorus3D obtained an initial solution in 456 milliseconds . It performed constraint satisfaction typically within 250 milliseconds to reflect the user 's dragging a graph node . 6.2 Constrained Dragging The second example is an application which allows a user to drag an object constrained to be on another spherical object . Figure 2 depicts this application , where the smaller solid spherical object is constrained to be on the surface of the larger wireframe one . The application declares a strong Euclidean geometric constraint which specifies a constant distance between the centers of these objects . When the user tries to drag the smaller object with a mouse , the application imposes another medium Euclidean constraint which collinearly locates the viewpoint , the 3D position of the mouse cursor -LRB- which is considered to be on the screen -RRB- , and 1 Unlike constrained dragging in the next example , this mouse operation is simply implemented with Java 3D 's PickMouseBehavior classes . Figure 2 : Dragging an object constrained to be on a sphere . Viewpoint Mouse cursor which is on the screen Screen Object which is on the sphere surface Collinearity constraint Distance constraint Sphere Figure 3 : Implementation of constrained dragging . the center of the dragged object as shown in Figure 3 . This collinearity constraint reflects the motion of the mouse in the position of the dragged object . Since the collinearity constraint is weaker than the first Euclidean constraint , the user can not drag the smaller object to the outside of the larger sphere . The application initially declares one Euclidean geometric constraint on two point variables , and solved it in 1 millisecond on the same computer as the first example . When the user tries to drag the smaller object , it adds another Euclidean constraint as well as two edit constraints for the viewpoint and mouse position . The solver maintained this constraint system usually within 2 milliseconds . 6.3 Inverse Kinematics The final example applies inverse kinematics to a virtual robot arm by using constraints . Unlike the previous examples , it takes advantage of coordinate transformations to express its constraint system . 98 -LRB- a -RRB- -LRB- b -RRB- -LRB- c -RRB- -LRB- d -RRB- -LRB- e -RRB- -LRB- f -RRB- Figure 4 : A robot arm application which performs inverse kinematics . As illustrated in Figure 4 -LRB- a -RRB- , the robot arm consists of four parts called a base , a shoulder , an upper arm , and a forearm . Constraint satisfaction for inverse kinematics is performed to position its hand -LRB- the end of the forearm -RRB- at the target object if possible , or otherwise to make it maximally close to the target . Figures 4 -LRB- b -RRB- -LRB- f -RRB- show the movement of the robot arm . In Figures 4 -LRB- b -RRB- -LRB- e -RRB- , its hand is positioned at the exact location of the target by using appropriate angles of its joints . By contrast , in Figure 4 -LRB- f -RRB- , the hand can not reach the target , and therefore the arm is extended toward the target instead . Figure 5 describes the constraint program used in the robot arm application . After constructing a constraint solver s , it creates six coordinate transformations shldrTTfm , shldrRTfm , uarmTTfm , uarmRTfm , farmTTfm , and farmRTfm . Here the rotation angle parameters of the rotation transformations shldrRTfm , uarmRTfm , and farmRTfm will actually work as variables that can be altered by the solver . Next , it generates a point variable handPos to represent the position of the hand , and then suggests the target position to the hand by using a preferential edit constraint editHandPos . Finally , executing the solver , it obtains the desired angles shldrAngle , uarmAngle , and farmAngle of the rotation transformations . These angles will be passed to the Java 3D library to render the properly configured robot arm . This program generates a constraint system which contains three translation and three rotation transformations , one explicit point variable as well as six point variables and three 1D variables for coordinate transformations , and one edit constraint . The solver found an initial solution to this system in 18 milliseconds , and obtained each new solution for a frame update typically within 10 milliseconds . There has been work on integrating constraints or similar functions with 3D graphics languages to facilitate the specification of graphical objects . For example , we can view the event routing mechanism in VRML -LSB- 4 -RSB- as a limited form of one-way propagation constraints . Also , there is an attempt to extend VRML by introducing one-way propagation and finite-domain combinatorial constraints -LSB- 5 -RSB- . However , they can not handle more powerful simultaneous nonlinear constraints such as Euclidean geometric constraints . Although many constraint solvers have been developed in 99 / / constraint solver s = new C3Solver -LRB- -RRB- ; / / translation transformation for the shoulder : fixed to -LRB- 0 , .1 , 0 -RRB- shldrTTfm = new C3TranslateTransform -LRB- new C3Domain3D -LRB- 0 , .1 , 0 -RRB- -RRB- ; s.add -LRB- shldrTTfm -RRB- ; / / shldrTTfm is parented by the world coordinate system / / rotation transformation for the shoulder : axis fixed to -LRB- 0 , 1 , 0 -RRB- ; angle ranging over -LSB- -10000 , 10000 -RSB- shldrRTfm = new C3RotateTransform -LRB- new C3Domain3D -LRB- 0 , 1 , 0 -RRB- , new C3Domain -LRB- -10000 , 10000 -RRB- -RRB- ; s.add -LRB- shldrRTfm , shldrTTfm -RRB- ; / / shldrRTfm is parented by shldrTTfm / / translation transformation for the upper arm : fixed to -LRB- 0 , .1 , 0 -RRB- uarmTTfm = new C3TranslateTransform -LRB- new C3Domain3D -LRB- 0 , .1 , 0 -RRB- -RRB- ; s.add -LRB- uarmTTfm , shldrRTfm -RRB- ; / / uarmTTfm is parented by shldrRTfm / / rotation transformation for the upper arm : axis fixed to -LRB- 0 , 0 , 1 -RRB- ; angle ranging over -LSB- -1.57 , 1.57 -RSB- uarmRTfm = new C3RotateTransform -LRB- new C3Domain3D -LRB- 0 , 0 , 1 -RRB- , new C3Domain -LRB- -1.57 , 1.57 -RRB- -RRB- ; s.add -LRB- uarmRTfm , uarmTTfm -RRB- ; / / uarmRTfm is parented by uarmTTfm / / translation transformation for the forearm : fixed to -LRB- 0 , .5 , 0 -RRB- farmTTfm = new C3TranslateTransform -LRB- new C3Domain3D -LRB- 0 , .5 , 0 -RRB- -RRB- ; s.add -LRB- farmTTfm , uarmRTfm -RRB- ; / / farmTTfm is parented by uarmRTfm / / rotation transformation for the forearm : axis fixed to -LRB- 0 , 0 , 1 -RRB- ; angle ranging over -LSB- -3.14 , 0 -RSB- farmRTfm = new C3RotateTransform -LRB- new C3Domain3D -LRB- 0 , 0 , 1 -RRB- , new C3Domain -LRB- -3.14 , 0 -RRB- -RRB- ; s.add -LRB- farmRTfm , farmTTfm -RRB- ; / / farmRTfm is parented by farmTTfm / / variable for the hand 's position , associated with farmRTfm and fixed to -LRB- 0 , .5 , 0 -RRB- handPos = new C3Variable3D -LRB- farmRTfm , new C3Domain3D -LRB- 0 , .5 , 0 -RRB- -RRB- ; / / medium-strength edit constraint for the hand 's position editHandPos = new C3EditConstraint -LRB- handPos , C3.MEDIUM -RRB- ; s.add -LRB- editHandPos -RRB- ; / / suggest the hand being located at the target 's position editHandPos.set -LRB- getTargetWorldCoordinates -LRB- -RRB- -RRB- ; / / solve the constraint system s.solve -LRB- -RRB- ; / / get solutions double shldrAngle = shldrRTfm.rotationAngle -LRB- -RRB- . value -LRB- -RRB- ; double uarmAngle = uarmRTfm.rotationAngle -LRB- -RRB- . value -LRB- -RRB- ; double farmAngle = farmRTfm.rotationAngle -LRB- -RRB- . value -LRB- -RRB- ; Figure 5 : Constraint program for the robot arm application . the field of graphical user interfaces -LSB- 3 , 7 , 11 , 12 , 13 , 17 , 18 -RSB- , most of them do not provide special treatment for 3D graphics . In general , the role of nonlinear geometric constraints is more important in 3D applications than in 2D interfaces . Most importantly , 3D graphics usually requires rotations of objects which are rarely used in 2D interfaces . The main reason is that we often equally treat all `` horizontal '' directions in a 3D space even if we may clearly distinguish them from `` vertical '' directions . Therefore , nonlinear constraint solvers are appropriate for 3D applications . In addition , coordinate transformations should be supported since they are typically used to handle rotations of objects . Gleicher proposed the differential approach -LSB- 8 , 9 -RSB- , which supports 3D geometric constraints and coordinate transformations . In a sense , it shares a motivation with Chorus3D ; in addition to support for 3D graphics , it allows user-defined kinds of geometric constraints . However , it is based on a different solution method from Chorus3D ; it realizes constraint satisfaction by running virtual dynamic simulations . This difference results in a quite different behavior of solutions as well as an interface for controlling solutions . By contrast , Chorus3D provides a much more compatible interface with recent successful solvers such as Cassowary -LSB- 3 -RSB- . Much research on inverse kinematics has been conducted in the fields of computer graphics and robotics -LSB- 1 , 20 -RSB- . However , inverse kinematics is typically implemented as specialized software which only provides limited kinds of geometric constraints . Chorus3D has two limitations in its algorithm : one is on the precision of solutions determined by preferential constraints ; the other is on the speed of the satisfaction of large constraint systems . These limitations are mainly caused by the treatment of multi-level preferences of constraints in addition to required constraints -LRB- i.e. , constraint hierarchies -RRB- . Although many numerical optimization techniques have been proposed and implemented in the field of mathematical programming -LSB- 2 , 6 -RSB- , most of them do not handle preferential constraints . To alleviate the limitations of Chorus3D , we are pursuing a more sophisticated method for processing multi-level preferential constraints . We implemented Chorus3D as a class library which can be exploited in C++ and Java programs . However , more high-level authoring tools will also be useful for declarative approaches to 3D design . One possible direction is to extend VRML -LSB- 4 -RSB- to support geometric constraints . Standard VRML requires scripts in Java or JavaScript to realize complex layouts and behaviors . By contrast , constraint-enabled VRML will cover a wider range of applications without such additional scripts . In this paper , we presented Chorus3D , a geometric constraint library for 3D graphical applications . It enables programmers to use geometric constraints for various purposes 100 such as geometric layout , constrained dragging , and inverse kinematics . Its novel feature is to handle scene graphs by processing coordinate transformations in geometric constraint satisfaction . Our future work includes the development of other kinds of geometric constraints to further prove the usefulness of our approach . In particular , we are planning to implement non-overlapping constraints -LSB- 13 -RSB- in Chorus3D so that we can use it for the collision resolution of graphical objects . Another future direction is to improve Chorus3D in the scalability and accuracy of constraint satisfaction . -LSB- 1 -RSB- Badler , N. I. , Phillips , C. B. , and Webber , B. L. Simulating Humans : Computer Graphics , Animation , and Control . Oxford University Press , Oxford , 1993 . -LSB- 2 -RSB- Bertsekas , D. P. Nonlinear Programming , 2nd ed . Athena Scientific , 1999 . -LSB- 3 -RSB- Borning , A. , Marriott , K. , Stuckey , P. , and Xiao , Y. Solving linear arithmetic constraints for user interface applications . In Proc . ACM UIST , 1997 , 87 96 . -LSB- 4 -RSB- Carey , R. , Bell , G. , and Marrin , C . The Virtual Reality Modeling Language -LRB- VRML97 -RRB- . ISO/IEC 14772-1:1997 , The VRML Consortium Inc. , 1997 . -LSB- 5 -RSB- Diehl , S. , and Keller , J. VRML with constraints . In Proc . Web3D-VRML , ACM , 2000 , 81 86 . -LSB- 6 -RSB- Fletcher , R. Practical Methods of Optimization , 2nd ed . John Wiley & Sons , 1987 . -LSB- 7 -RSB- Freeman-Benson , B. N. , Maloney , J. , and Borning , A . An incremental constraint solver . Commun . ACM 33 , 1 -LRB- 1990 -RRB- , 54 63 . -LSB- 8 -RSB- Gleicher , M . A graphical toolkit based on differential constraints . In Proc . ACM UIST , 1993 , 109 120 . -LSB- 9 -RSB- Gleicher , M . A differential approach to graphical manipulation -LRB- Ph.D. thesis -RRB- . Tech . Rep. CMU-CS-94-217 , Sch . Comput . Sci . Carnegie Mellon Univ. , 1994 . -LSB- 10 -RSB- Herrera , F. , Lozano , M. , and Verdegay , J. L. Tackling real-coded genetic algorithms : Operators and tools for behavioural analysis . Artif . Intell . Rev. 12 , 4 -LRB- 1998 -RRB- , 265 319 . -LSB- 11 -RSB- Heydon , A. , and Nelson , G . The Juno-2 constraint-based drawing editor . Research Report 131a , Digital Systems Research Center , 1994 . -LSB- 12 -RSB- Hosobe , H . A scalable linear constraint solver for user interface construction . In Principles and Practice of Constraint Programming -- CP2000 , vol . 1894 of LNCS , Springer , 2000 , 218 232 . -LSB- 13 -RSB- Hosobe , H . A modular geometric constraint solver for user interface applications . In Proc . ACM UIST , 2001 , 91 100 . -LSB- 14 -RSB- Kamada , T. , and Kawai , S . An algorithm for drawing general undirected graphs . Inf . Process . Lett . 31 , 1 -LRB- 1989 -RRB- , 7 15 . -LSB- 15 -RSB- Kitano , H. , Ed . Genetic Algorithms . Sangyo-Tosho , 1993 . In Japanese . -LSB- 16 -RSB- Kramer , G. A . A geometric constraint engine . Artif . Intell . 58 , 1 3 -LRB- 1992 -RRB- , 327 360 . -LSB- 17 -RSB- Marriott , K. , Chok , S. S. , and Finlay , A . A tableau based constraint solving toolkit for interactive graphical applications . In Principles and Practice of Constraint Programming -- CP98 , vol . 1520 of LNCS , Springer , 1998 , 340 354 . -LSB- 18 -RSB- Sannella , M. Skyblue : A multi-way local propagation constraint solver for user interface construction . In Proc . ACM UIST , 1994 , 137 146 . -LSB- 19 -RSB- Takahashi , S. Visualizing constraints in visualization rules . In Proc . CP2000 Workshop on Analysis and Visualization of Constraint Programs and Solvers , 2000 . -LSB- 20 -RSB- Zhao , J. , and Badler , N. I. Inverse kinematics positioning using nonlinear programming for highly articulated figures . ACM Trans . Gr . 13 , 4 -LRB- 1994 -RRB- , 313 336 . 101

O O O O O B I I O O O O O O O B I O O O O O O O O O O O B I I O O O O O O O O O O O O B O O O O B I O B I O O O O O O O O O O O O B O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I I O O O O O O B I O O O O O B I O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O B I I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

coordinate transformation
3d graphical applications
3d graphics
scene graphs
geometric layout
geometric constraints
graphical objects
constraint satisfaction
layout
behaviors