Use

of

Relative

Code

Churn

Measures

to

Predict

System

Defect

Density

Software

systems

evolve

over

time

due

to

changes

in

requirements

optimization

of

code

fixes

for

security

and

reliability

bugs

etc.

Code

churn

which

measures

the

changes

made

to

a

component

over

a

period

of

time

quantifies

the

extent

of

this

change

We

present

a

technique

for

early

prediction

of

system

defect

density

using

a

set

of

relative

code

churn

measures

that

relate

the

amount

of

churn

to

other

variables

such

as

component

size

and

the

temporal

extent

of

churn

Using

statistical

regression

models

we

show

that

while

absolute

measures

of

code

churn

are

poor

predictors

of

defect

density

our

set

of

relative

measures

of

code

churn

is

highly

predictive

of

defect

density

A

case

study

performed

on

Windows

Server

indicates

the

validity

of

the

relative

code

churn

measures

as

early

indicators

of

system

defect

density

Furthermore

our

code

churn

metric

suite

is

able

to

discriminate

between

fault

and

not

fault-prone

binaries

with

an

accuracy

of

percent

D.

-LSB-

Software

Engineering

-RSB-

Distribution

Maintenance

and

Enhancement

Version

control

D.

-LSB-

Software

Engineering

-RSB-

Metrics

Performance

measures

Process

metrics

Product

metrics

Measurement

Design

Reliability

A

reliability

chasm

often

separates

the

quality

of

a

software

product

observed

in

its

pre-release

testing

in

a

software

development

shop

and

its

post-release

use

in

the

field

That

is

true

field

reliability

as

measured

by

the

number

of

failures

found

by

customers

over

a

period

of

time

can

not

be

measured

before

a

product

has

been

completed

and

delivered

to

a

customer

Because

true

reliability

information

is

available

late

in

the

process

corrective

actions

tend

to

be

expensive

-LSB-

-RSB-

Clearly

software

organizations

can

benefit

in

many

ways

from

an

early

warning

system

concerning

potential

post-release

defects

in

their

product

to

guide

corrective

actions

to

the

quality

of

the

software

We

use

code

churn

to

predict

the

defect

density

in

software

systems

Code

churn

is

a

measure

of

the

amount

of

code

change

taking

place

within

a

software

unit

over

time

It

is

easily

extracted

from

a

system

's

change

history

as

recorded

automatically

by

a

version

control

system

Most

version

control

systems

use

a

file

comparison

utility

-LRB-

such

as

diff

-RRB-

to

automatically

estimate

how

many

lines

were

added

deleted

and

changed

by

a

programmer

to

create

a

new

version

of

a

file

from

an

old

version

These

differences

are

the

basis

of

churn

measures

We

create

and

validate

a

set

of

relative

code

churn

measures

as

early

indicators

of

system

defect

density

Relative

churn

measures

are

normalized

values

of

the

various

measures

obtained

during

the

churn

process

Some

of

the

normalization

parameters

are

total

lines

of

code

file

churn

file

count

etc.

Munson

et

al.

-LSB-

-RSB-

use

a

similar

relative

approach

towards

establishing

a

baseline

while

studying

code

churn

Studies

have

shown

that

absolute

measures

like

LOC

are

poor

predictors

of

pre

and

post

release

faults

-LSB-

-RSB-

in

industrial

software

systems

In

general

process

measures

based

on

change

history

have

been

found

be

better

indicators

of

fault

rates

than

product

metrics

of

code

-LSB-

-RSB-

In

an

evolving

system

it

is

highly

beneficial

to

use

a

relative

approach

to

quantify

the

change

in

a

system

As

we

show

these

relative

measures

can

be

devised

to

cross

check

each

other

so

that

the

metrics

do

not

provide

conflicting

information

Our

basic

hypothesis

is

that

code

that

changes

many

times

pre-release

will

likely

have

more

post-release

defects

than

code

that

changes

less

over

the

same

period

of

time

More

precisely

we

address

the

hypotheses

shown

in

Table

Our

experiments

on

Windows

Server

-LRB-

-RRB-

support

these

four

hypotheses

with

high

statistical

significance

We

analyzed

the

code

churn

between

the

release

of

and

the

release

of

the

Service

Pack

-LRB-

-RRB-

to

predict

the

defect

density

in

The

relative

code

churn

measures

are

statistically

better

predictors

of

defect

density

than

the

absolute

measures

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

ICSE

May

St.

Louis

MO

USA

Copyright

ACM

_________________________________________________________________________________

Nachiappan

Nagappan

was

an

intern

with

the

Testing

Verification

and

Measurement

Group

Microsoft

Research

in

the

Summer

of

when

this

work

was

carried

out

They

also

they

are

indicative

of

increase

in

system

defect

density

and

can

accurately

predict

the

system

defect

density

with

a

high

degree

of

sensitivity

Our

metric

suite

is

able

to

discriminate

between

fault

and

not

fault-prone

binaries

in

with

an

accuracy

of

percent

Table

Research

Hypotheses

Hypothesis

H

Increase

in

relative

code

churn

measures

is

accompanied

by

an

increase

in

system

defect

density

H

Using

relative

values

of

code

churn

predictors

is

better

than

using

direct

-LRB-

absolute

-RRB-

values

to

explain

the

system

defect

density

H

Relative

code

churn

measures

can

be

used

as

efficient

predictors

of

system

defect

density

H

Relative

code

churn

measures

can

be

used

to

discriminate

between

fault

and

not

fault-prone

binaries

The

organization

of

this

paper

is

as

follows

Section

describes

the

related

work

Section

explains

data

collection

and

section

the

relative

code

churn

measures

Section

presents

the

case

study

and

the

observed

results

Section

discusses

our

conclusions

and

future

work

Prior

analyses

on

predicting

defect

density

used

code

churn

measures

as

part

of

a

larger

set

of

metrics

Code

churn

measures

have

not

been

studied

in

isolation

as

predictors

of

software

defect

density

The

background

work

presented

below

is

from

studies

that

involved

industrial

software

systems

The

source

code

base

of

is

two

orders

of

magnitude

larger

than

the

largest

example

considered

below

Munson

et

al.

-LSB-

-RSB-

observe

that

as

a

system

is

developed

the

relative

complexity

of

each

program

module

that

has

been

altered

-LRB-

or

churned

-RRB-

also

will

change

The

rate

of

change

in

relative

complexity

serves

as

a

good

index

of

the

rate

of

fault

injection

They

studied

a

KLOC

-LRB-

thousand

lines

of

code

-RRB-

embedded

real

time

system

with

modules

programmed

in

C.

Code

churn

metrics

were

found

to

be

among

the

most

highly

correlated

with

problem

reports

-LSB-

-RSB-

Khoshgoftaar

et

al.

-LSB-

-RSB-

define

debug

churn

as

the

number

of

lines

of

code

added

or

changed

for

bug

fixes

Their

objective

was

to

identify

modules

where

debug

code

churn

exceeds

a

threshold

in

order

to

classify

the

modules

as

fault-prone

They

studied

two

consecutive

releases

of

a

large

legacy

system

for

telecommunications

The

system

contained

over

procedures

in

modules

Discriminant

analysis

identified

fault-prone

modules

based

on

static

software

product

metrics

Their

model

when

used

on

the

second

release

showed

a

type

I

and

II

misclassification

rate

of

respectively

and

an

overall

misclassification

rate

of

Ohlsson

et

al.

-LSB-

-RSB-

identify

fault-prone

modules

by

analyzing

legacy

software

through

successive

releases

They

use

a

total

of

measures

twelve

of

which

are

based

on

size

and

change

measures

These

measures

were

used

to

identify

percent

of

the

most

fault-prone

components

successfully

Karunanithi

-LSB-

-RSB-

uses

a

neural

network

approach

for

software

reliability

growth

modeling

in

the

presence

of

continuous

code

churn

which

he

shows

improves

over

the

traditional

time-domain

based

models

Similarly

Khoshgoftaar

et

al.

-LSB-

-RSB-

use

code

churn

as

a

measure

of

software

quality

in

a

program

of

lines

of

assembly

language

Using

eight

complexity

measures

including

code

churn

they

found

neural

networks

and

multiple

regression

to

be

an

efficient

predictor

of

software

quality

as

measured

by

gross

change

in

the

code

They

suggest

that

using

neural

networks

may

not

work

in

all

environments

and

the

results

obtained

are

environment

specific

Neural

networks

can

be

used

for

improving

software

maintenance

-LSB-

-RSB-

Ostrand

et

al.

-LSB-

-RSB-

use

information

of

file

status

such

as

new

changed

unchanged

files

along

with

other

explanatory

variables

such

as

lines

of

code

age

prior

faults

etc.

as

predictors

in

a

negative

binomial

regression

equation

to

predict

the

number

of

faults

in

a

multiple

release

software

system

The

predictions

made

using

binomial

regression

model

were

of

a

high

accuracy

for

faults

found

in

both

early

and

later

stages

of

development

-LSB-

-RSB-

Closely

related

to

our

study

is

the

work

performed

by

Graves

et

al.

-LSB-

-RSB-

on

predicting

fault

incidences

using

software

change

history

Several

statistical

models

were

built

based

on

a

weighted

time

damp

model

using

the

sum

of

contributions

from

all

changes

to

a

module

in

its

history

The

most

successful

model

computes

the

fault

potential

by

summing

contributions

from

changes

to

the

module

where

large

and/or

recent

changes

contribute

the

most

to

fault

potential

-LSB-

-RSB-

This

is

similar

to

our

approach

of

using

relative

measures

to

predict

fault

potential

Drawing

general

conclusions

from

empirical

studies

in

software

engineering

is

difficult

because

any

process

depends

to

a

large

degree

on

a

potentially

large

number

of

relevant

context

variables

For

this

reason

we

can

not

assume

a

priori

that

the

results

of

a

study

generalize

beyond

the

specific

environment

in

which

it

was

conducted

-LSB-

-RSB-

Researchers

become

more

confident

in

a

theory

when

similar

findings

emerge

in

different

contexts

-LSB-

-RSB-

Towards

this

end

we

intend

that

our

case

study

contributes

towards

strengthening

the

existing

empirical

body

of

knowledge

in

this

field

-LSB-

-RSB-

The

baseline

used

for

measuring

the

code

churn

and

other

measures

described

below

is

Windows

Server

-LRB-

-RRB-

We

measured

churn

between

this

baseline

and

Windows

Server

Service

Pack

-LRB-

-RRB-

We

sometimes

refer

to

as

the

new

version

of

the

code

Service

packs

are

a

means

by

which

product

updates

are

distributed

Service

packs

contain

updates

for

system

reliability

program

compatibility

security

etc.

that

are

conveniently

bundled

for

easy

downloading

The

size

of

the

code

base

analyzed

is

million

LOC

-LRB-

KLOC

-RRB-

This

consisted

of

binaries

which

were

compiled

from

files

Some

files

contribute

to

more

than

one

binary

As

defects

for

are

reported

at

the

binary

level

we

relate

churn

to

defects

at

the

level

of

binaries

The

absolute

measures

and

methods

of

data

collection

are

described

below

Total

LOC

is

the

number

of

lines

of

non-commented

executable

lines

in

the

files

comprising

the

new

version

of

a

binary

Internal

Microsoft

tools

were

used

to

compute

this

measure

Churned

LOC

is

the

sum

of

the

added

and

changed

lines

of

code

between

a

baseline

version

and

a

new

version

of

the

files

comprising

a

binary

Deleted

LOC

is

the

number

of

lines

of

code

deleted

between

the

baseline

version

and

the

new

version

of

a

binary

The

churned

LOC

and

the

deleted

LOC

are

computed

by

the

version

control

systems

using

a

file

comparison

utility

like

diff

File

count

is

the

number

of

files

compiled

to

create

a

binary

Weeks

of

churn

is

the

cumulative

time

that

a

file

was

opened

for

editing

from

the

version

control

system

Churn

count

is

the

number

of

changes

made

to

the

files

comprising

a

binary

between

the

two

versions

-LRB-

and

-RRB-

Files

churned

is

the

number

of

files

within

the

binary

that

churned

In

this

section

we

describe

our

relative

code

churn

measures

The

churn

measures

are

denoted

by

the

elements

The

elements

and

their

relationship

to

defect

density

are

explained

below

-LRB-

these

relationships

are

verified

in

section

-RRB-

Churned

LOC

Total

LOC

We

expect

the

larger

the

proportion

of

churned

-LRB-

added

changed

-RRB-

code

to

the

LOC

of

the

new

binary

the

larger

the

magnitude

of

the

defect

density

for

that

binary

will

be

Deleted

LOC

Total

LOC

We

expect

the

larger

the

proportion

of

deleted

code

to

the

LOC

of

the

new

binary

the

larger

the

magnitude

of

the

defect

density

for

that

binary

will

be

Files

churned

File

count

We

expect

the

greater

the

proportion

of

files

in

a

binary

that

get

churned

the

greater

the

probability

of

these

files

introducing

defects

For

e.g.

suppose

binaries

A

and

B

contain

twenty

files

each

If

binary

A

has

five

churned

files

and

binary

B

has

two

churned

files

we

expect

binary

A

to

have

a

higher

defect

density

Churn

count

Files

churned

Suppose

binaries

A

and

B

have

twenty

files

each

and

also

have

five

churned

files

each

If

the

five

files

in

binary

A

are

churned

twenty

times

and

the

five

files

in

binary

B

are

churned

ten

times

then

we

expect

binary

A

to

have

a

higher

defect

density

acts

as

a

cross

check

on

Weeks

of

churn

File

count

is

used

to

account

for

the

temporal

extent

of

churn

A

higher

value

of

indicates

that

it

took

a

longer

time

to

fix

a

smaller

number

of

files

This

may

indicate

that

the

binary

contains

complex

files

that

may

be

hard

to

modify

correctly

Thus

we

expect

that

an

increase

in

would

be

accompanied

by

an

increase

in

the

defect

density

of

the

related

binary

Lines

worked

on

Weeks

of

churn

The

measure

Lines

worked

on

is

the

sum

of

the

churned

LOC

and

the

deleted

LOC

measures

the

extent

of

code

churn

over

time

in

order

to

cross

check

on

Weeks

of

churn

does

not

necessarily

indicate

the

amount

of

churn

reflects

our

expectation

that

the

more

lines

are

worked

on

the

longer

the

weeks

of

churn

should

be

A

high

value

of

cross

checks

on

and

should

predict

a

higher

defect

density

Churned

LOC

Deleted

LOC

is

used

in

order

to

quantify

new

development

All

churn

is

not

due

to

bug

fixes

In

feature

development

the

lines

churned

is

much

greater

than

the

lines

deleted

so

a

high

value

of

indicates

new

feature

development

acts

as

a

cross

check

on

and

neither

of

which

accurately

predicts

new

feature

development

Lines

worked

on

Churn

count

We

expect

that

the

larger

a

change

-LRB-

lines

worked

on

-RRB-

relative

to

the

number

of

changes

-LRB-

churn

count

-RRB-

the

greater

the

defect

density

will

be

acts

as

a

cross

check

on

and

as

well

as

and

With

respect

to

and

measures

the

amount

of

actual

change

that

took

place

cross

checks

to

account

for

the

fact

that

files

are

not

getting

churned

repeatedly

for

small

fixes

also

cross

checks

on

and

to

account

for

the

fact

that

the

higher

the

value

of

-LRB-

more

lines

per

churn

-RRB-

the

higher

is

the

time

-LRB-

-RRB-

and

lines

worked

on

per

week

-LRB-

-RRB-

-RRB-

If

this

is

not

so

then

a

large

amount

of

churn

might

have

been

performed

in

a

small

amount

of

time

which

can

cause

an

increased

defect

density

Figure

illustrates

the

cross

check

relationships

of

these

relative

code

churn

measures

As

discussed

above

and

cross

check

on

each

other

and

cross

checks

on

the

set

of

and

All

these

measures

triangulate

on

their

respective

dependent

measures

with

the

goal

of

providing

the

best

possible

estimate

of

defect

density

with

a

minimum

inflation

in

the

estimation

We

now

describe

the

case

study

performed

at

Microsoft

Section

presents

the

correlation

analysis

between

the

relative

code

churn

measures

and

system

defect

density

Section

details

the

model

building

activities

and

Section

the

predictive

ability

of

the

models

Section

discusses

the

discriminative

power

of

the

relative

code

churn

measures

and

Section

the

limitations

of

the

study

Figure

Relative

Churn

Measure

Cross

Check

Relationships

Table

Cross

Correlations

All

correlations

are

significant

at

the

-LRB-

-RRB-

level

-LRB-

-RRB-

Defects

KLOC

Defects

KLOC

As

mentioned

before

the

system

defect

density

for

was

collected

at

the

level

of

binaries

That

is

for

each

binary

we

have

a

count

of

the

number

of

defects

assigned

to

that

binary

Throughout

the

rest

of

the

paper

we

assume

a

statistical

significance

at

confidence

-LRB-

level

of

significance

-LRB-

-RRB-

-RRB-

Correlation

Analysis

Our

goal

is

to

verify

that

with

an

increase

in

the

code

churn

measures

-LRB-

-RRB-

there

is

a

statistically

significant

increase

in

the

defects/KLOC

Table

shows

the

Spearman

rank

correlation

-LRB-

-RRB-

among

the

defects/KLOC

and

the

relative

code

churn

measures

Spearman

rank

correlation

is

a

commonly-used

robust

correlation

technique

-LSB-

-RSB-

because

it

can

be

applied

even

when

the

association

between

elements

is

non-linear

Table

shows

that

there

exists

a

statistically

significant

-LRB-

at

confidence

-RRB-

positive

relationship

between

the

measures

and

the

defects/KLOC

-LRB-

shown

in

bold

-RRB-

Thus

with

an

increase

in

the

relative

churn

measures

there

is

a

corresponding

positive

increase

in

the

defects/KLOC

This

is

indicated

by

the

statistically

significant

positive

Spearman

rank

correlation

coefficient

From

the

above

observations

we

conclude

that

an

increase

in

relative

code

churn

measures

is

accompanied

by

an

increase

in

system

defect

density

-LRB-

H

-RRB-

In

order

to

illustrate

the

cross

checks

better

consider

the

measures

and

in

Figure

with

their

Spearman

rank

correlation

coefficients

from

Table

Figure

Cross

Correlation

Relationships

The

Spearman

correlation

coefficient

of

between

and

indicates

that

there

is

a

very

strong

correlation

between

the

two

measures

But

this

might

not

be

the

case

when

there

is

a

higher

proportion

of

churned

code

compared

to

deleted

code

-LRB-

as

measured

by

for

new

feature

development

-RRB-

Since

this

can

not

be

measured

by

or

acts

as

a

cross

check

on

them

The

correlation

between

and

-LRB-

-RRB-

indicates

when

there

is

a

Churned

LOC

Total

LOC

Deleted

LOC

Total

LOC

Files

churned

File

count

Churn

count

Files

churned

Weeks

of

churn

File

count

Lines

worked

on

Weeks

of

churn

Churned

LOC

Deleted

LOC

Lines

worked

on

Churn

count

Cross

check

new

feature

addition

there

is

a

corresponding

increase

in

the

churned

code

For

and

this

correlation

is

not

as

strong

-LRB-

but

is

statistically

significant

-RRB-

because

there

were

relatively

fewer

new

feature

additions

compared

to

other

changes

in

the

source

base

Model

Fitting

We

now

compare

predictive

models

built

using

absolute

measures

against

those

built

using

the

relative

churn

measures

For

the

absolute

model

defects/KLOC

is

the

dependent

variable

and

the

predictors

are

the

absolute

measures

described

in

Section

For

the

relative

model

defects/KLOC

is

the

dependent

variable

and

the

predictors

are

the

relative

measures

described

in

Section

R

is

a

measure

of

variance

in

the

dependent

variable

that

is

accounted

for

by

the

model

built

using

the

predictors

-LSB-

-RSB-

R

is

a

measure

of

the

fit

for

the

given

data

set

-LRB-

It

can

not

be

interpreted

as

the

quality

of

the

dataset

to

make

future

predictions

-RRB-

The

adjusted

R

measure

also

can

be

used

to

evaluate

how

well

a

model

will

fit

a

given

data

set

-LSB-

-RSB-

Adjusted

R

explains

for

any

bias

in

the

R

measure

by

taking

into

account

the

degrees

of

freedom

of

the

predictor

variables

and

the

sample

population

The

adjusted

R

tends

to

remain

constant

as

the

R

measure

for

large

population

samples

The

multiple

regression

model

fit

for

absolute

measures

using

all

the

predictors

has

an

R

value

of

-LRB-

F

p

-RRB-

-LRB-

The

F-ratio

is

used

to

test

the

hypothesis

that

all

regression

coefficients

are

zero

-RRB-

This

is

a

poor

fit

of

the

data

and

irrespective

of

other

transformations

-LRB-

like

for

e.g.

log

-RRB-

we

can

not

get

a

marked

improvement

in

R

The

adjusted

R

value

for

the

absolute

measures

is

Throughout

the

rest

of

this

paper

we

present

the

adjusted

R

values

in

addition

to

the

R

measures

in

order

to

eliminate

any

bias

in

model

building

But

with

respect

to

the

large

sample

size

-LRB-

binaries

-RRB-

the

adjusted

R

and

R

value

show

only

minor

variation

not

sufficient

enough

to

drop

the

R

value

and

employ

the

adjusted

R

value

There

are

different

ways

in

which

regression

models

-LSB-

-RSB-

can

be

built

Three

common

regression

methods

-LSB-

-RSB-

are

forward

backward

and

step-wise

regression

In

forward

regression

one

adds

a

single

predictor

at

a

time

to

the

model

based

on

the

strength

of

its

correlation

with

the

dependent

variable

The

effect

of

adding

each

predictor

is

evaluated

based

on

the

results

of

an

F-ratio

test

-LSB-

-RSB-

Variables

that

do

not

significantly

add

to

the

success

of

the

model

are

excluded

In

backward

regression

a

model

is

built

using

all

the

predictors

The

weakest

predictor

variable

is

removed

and

the

strength

of

the

overall

built

model

is

assessed

similar

to

the

forward

regression

procedure

If

this

significantly

weakens

the

model

then

the

predictor

is

put

back

-LRB-

and

otherwise

removed

-RRB-

Step-wise

regression

-LSB-

-RSB-

is

the

more

robust

technique

of

these

methods

The

initial

model

consists

of

the

predictor

having

the

single

largest

correlation

with

the

dependent

variable

Subsequently

new

predictors

are

selected

for

addition

into

the

model

based

on

their

partial

correlation

with

the

predictors

already

in

the

model

With

each

new

set

of

predictors

the

model

is

evaluated

and

predictors

that

do

not

significantly

contribute

towards

statistical

significance

in

terms

of

the

F-ratio

are

removed

so

that

in

the

end

the

best

set

of

predictors

explaining

the

maximum

possible

variance

is

left

A

step-wise

regression

analysis

using

the

absolute

set

of

predictors

does

not

lead

to

any

significant

change

in

the

R

values

-LRB-

-RRB-

-LRB-

adjusted

R

-RRB-

Only

the

LOC

and

the

number

of

times

a

file

is

churned

are

kept

as

predictors

This

further

confirms

the

fact

that

using

the

absolute

measures

is

not

an

appropriate

method

for

assessing

the

system

defect

density

Several

empirical

studies

use

Principal

Component

Analysis

-LRB-

PCA

-RRB-

-LSB-

-RSB-

to

build

regression

models

-LSB-

-RSB-

In

PCA

a

smaller

number

of

uncorrelated

linear

combinations

of

metrics

which

account

for

as

much

sample

variance

as

possible

are

selected

for

use

in

regression

PCA

is

not

a

possible

solution

when

using

absolute

measures

because

the

correlation

matrix

is

not

positive

definite

We

still

use

the

two

principal

components

generated

to

build

a

multiple

regression

equation

The

multiple

regression

equation

constructed

has

an

even

lower

value

of

R

-LRB-

F

p

-RRB-

Based

on

the

three

results

discussed

above

-LRB-

multiple

regression

using

all

the

predictors

step-wise

regression

and

PCA

-RRB-

we

conclude

that

the

absolute

measures

are

not

good

predictors

of

system

defect

density

As

outlined

in

Section

we

calculate

the

relative

code

churn

measures

-LRB-

-RRB-

and

build

regression

models

using

all

the

measures

step-wise

regression

and

PCA

Table

shows

the

R

value

of

the

regression

equation

built

using

all

the

measures

We

also

present

the

adjusted

R

value

and

the

root

MSE

-LRB-

Mean

Squared

Error

-RRB-

Table

Regression

Fit

Using

All

Measures

Model

R

Adjusted

R

Root

MSE

All

Measures

Table

shows

how

the

R

value

changes

in

step-wise

regression

for

all

the

models

built

during

that

process

In

the

step-wise

regression

model

the

measure

is

dropped

The

best

R

value

in

Table

-LRB-

without

-RRB-

is

the

same

as

that

of

Table

-LRB-

-RRB-

but

there

is

a

change

in

the

third

decimal

place

of

the

standard

error

of

the

estimate

probably

was

dropped

because

there

were

relatively

fewer

new

feature

additions

compared

to

other

changes

in

the

source

base

The

adjusted

R

values

are

also

shown

but

are

not

significantly

different

from

the

R

values

due

to

the

large

sample

size

used

to

build

the

models

Table

Step-wise

Regression

Models

Model

R-Square

Adjusted

R-Square

Root

MSE

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

-LRB-

f

-RRB-

-LRB-

g

-RRB-

a

Predictors

-LRB-

Constant

-RRB-

b

Predictors

-LRB-

Constant

-RRB-

c

Predictors

-LRB-

Constant

-RRB-

d

Predictors

-LRB-

Constant

-RRB-

e

Predictors

-LRB-

Constant

-RRB-

f

Predictors

-LRB-

Constant

-RRB-

g

Predictors

-LRB-

Constant

-RRB-

The

PCA

of

the

eight

relative

code

churn

measures

yields

three

principal

components

PCA

can

account

for

the

multicollinearity

among

the

measures

which

can

lead

to

inflated

variance

in

the

estimation

of

the

defect

density

But

for

PCA

to

be

applicable

the

KMO

-LRB-

Kaiser-Meyer-Olkin

-RRB-

measure

-LSB-

-RSB-

of

sampling

adequacy

should

be

greater

than

-LSB-

-RSB-

The

KMO

measure

of

sampling

adequacy

is

a

test

of

the

amount

of

variance

within

the

data

that

can

be

explained

by

the

measures

The

KMO

measure

of

the

eight

relative

code

churn

measures

is

which

indicates

that

PCA

might

not

be

an

appropriate

method

to

apply

We

still

perform

the

analysis

to

investigate

and

present

those

results

as

well

on

a

comparative

basis

The

results

for

all

three

models

are

summarized

in

Table

Table

Relative

Measures

Model

Fits

Model

R

Adjusted

R

F-Test

sig

All

measures

-LRB-

p

-RRB-

Step-wise

regression

-LRB-

p

-RRB-

PCA

-LRB-

p

-RRB-

From

the

above

results

we

can

see

that

using

relative

values

of

code

churn

predictors

is

better

than

using

absolute

values

to

explain

the

system

defect

density

-LRB-

H

-RRB-

Figure

Actual

vs.

Estimated

System

Defect

Density

Defect

Density

Prediction

We

use

the

technique

of

data

splitting

-LSB-

-RSB-

to

measure

the

ability

of

the

relative

code

churn

measures

to

predict

system

defect

density

The

data

splitting

technique

was

employed

to

get

an

independent

assessment

of

how

well

the

defect

density

can

be

estimated

from

a

population

sample

We

randomly

select

two

thirds

of

the

binaries

-LRB-

-RRB-

to

build

the

prediction

model

and

use

the

remaining

one

third

-LRB-

-RRB-

to

verify

the

prediction

accuracy

We

constructed

models

using

all

the

measures

step-wise

regression

and

PCA

-LRB-

for

purpose

of

completeness

-RRB-

Table

shows

the

results

for

these

models

Table

Regression

Data

Fit

Model

R

Adjusted

R

F-Test

sig

All

measures

-LRB-

p

-RRB-

Step-wise

regression

-LRB-

dropped

-RRB-

-LRB-

p

-RRB-

PCA

-LRB-

p

-RRB-

Using

the

fitted

regression

equation

we

estimate

the

system

defect

density

for

the

remaining

binaries

Figure

shows

the

estimated

and

actual

defect

density

using

the

regression

equation

constructed

using

all

the

measures

-LRB-

sorted

by

estimated

defect

density

-RRB-

The

estimated

defect

density

is

shown

by

the

thicker

continuous

line

From

the

graph

we

can

see

that

the

estimated

defect

density

is

similar

to

the

actual

defect

density

The

axes

on

the

graphs

are

removed

in

order

to

protect

proprietary

data

To

quantify

the

sensitivity

of

prediction

we

run

a

correlation

analysis

between

the

estimated

and

actual

values

A

high

positive

correlation

coefficient

indicates

that

with

an

increase

in

the

actual

defect

density

there

is

a

corresponding

positive

increase

in

the

estimated

defect

density

We

perform

Pearson

and

Spearman

correlations

to

indicate

their

sensitivity

The

Pearson

correlation

indicates

a

linear

relationship

The

Spearman

correlation

is

a

more

robust

correlation

technique

Table

shows

that

the

correlations

are

all

positive

and

statistically

significant

The

magnitude

of

the

correlations

indicates

the

sensitivity

of

the

predictions

-LRB-

the

stronger

the

correlations

the

more

sensitive

are

the

predictions

-RRB-

The

models

built

using

all

the

measures

and

the

step-wise

method

have

the

same

sensitivity

and

are

better

than

the

model

built

using

PCA

Table

Correlation

Results

Model

Pearson

-LRB-

sig

-RRB-

Spearman

-LRB-

sig

-RRB-

All

measures

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Step-wise

regression

-LRB-

p

-RRB-

-LRB-

p

-RRB-

PCA

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Analyses

that

are

based

on

a

single

dataset

that

use

the

same

data

to

both

estimate

the

model

and

to

assess

its

performance

can

lead

to

unreasonably

negative

biased

estimates

of

sampling

variability

In

order

to

address

this

we

repeat

the

random

sampling

with

different

random

samples

to

verify

if

the

above

results

are

repeatable

For

each

sample

the

model

is

fit

with

binaries

to

build

the

model

Table

shows

the

fit

of

the

various

models

built

for

each

sample

Table

Random

Splits

Data

Fit

Model

R

Adjusted

R

F-Test

-LRB-

Sig

-RRB-

Random

All

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

drop

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

Random

All

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

drop

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

Random

All

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

drop

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

Using

each

of

the

above

predictive

models

we

calculate

the

estimated

defect

density

for

the

remaining

binaries

Table

shows

the

correlation

between

the

estimated

and

the

actual

defect

density

Table

Correlation

Between

Actual

and

Estimated

Defects/KLOC

Model

Pearson

Correlation

-LRB-

sig

-RRB-

Spearman

Correlation

-LRB-

sig

-RRB-

Random

All

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

All

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

All

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

Stepwise

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Random

PCA

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Based

on

the

consistent

positive

and

statistically

significant

correlations

indicating

the

sensitivity

of

predictions

obtained

in

Table

we

can

say

that

relative

code

churn

measures

can

be

used

as

efficient

predictors

of

system

defect

density

-LRB-

H

-RRB-

Our

results

demonstrate

it

is

effective

to

use

all

eight

measures

rather

than

dropping

any

of

them

from

the

predictive

equation

Each

of

these

measures

cross

check

on

each

other

and

any

abnormal

behavior

in

one

of

the

measures

-LRB-

for

e.g.

like

a

file

getting

churned

too

many

times

-RRB-

would

be

immediately

highlighted

By

interchanging

the

measures

in

a

model

equation

we

can

get

estimated

values

for

all

the

relative

measures

independently

For

example

in

order

to

determine

the

maximum

allowable

code

churn

with

respect

to

the

file

size

-LRB-

i.e.

-RRB-

say

for

a

particular

software

model

we

fix

the

maximum

allowable

system

defect

density

We

then

can

build

a

regression

model

with

and

defect

density

as

predictors

and

as

the

dependent

variable

Discriminant

Analysis

Discriminant

analysis

is

a

statistical

technique

used

to

categorize

programs

into

groups

based

on

the

metric

values

It

has

been

used

as

a

tool

for

the

detection

of

fault-prone

programs

-LSB-

-RSB-

The

ANSI-IEEE

Std

-LSB-

-RSB-

defines

a

fault

as

an

accidental

condition

that

causes

a

functional

unit

to

fail

to

perform

its

required

function

We

use

discriminant

analysis

to

identify

binaries

as

fault-prone

or

not

fault-prone

To

classify

if

a

binary

is

fault-prone

or

not

we

use

the

system

defect

density

in

a

normal

confidence

interval

calculation

as

shown

in

equation

LB

x

z

Standard

deviation

of

defect

density

-LRB-

-RRB-

n

where

LB

is

the

lower

bound

on

system

defect

density

x

is

the

mean

of

defect

density

Z

is

the

upper

quantile

of

the

standard

normal

distribution

n

is

the

number

of

observations

We

conservatively

classify

all

binaries

that

have

a

defect

density

less

than

LB

as

not

fault-prone

and

the

remaining

as

fault-prone

Table

shows

the

eigenvalue

and

overall

classification

ability

of

using

the

eight

measures

and

the

three

principal

components

The

eigenvalue

is

a

measure

of

the

discriminative

ability

of

the

discriminant

function

The

higher

the

eigenvalue

the

better

is

the

discriminative

ability

For

all

measures

the

function

correctly

classifies

nearly

nine

out

of

every

ten

binaries

Table

Overall

Discriminant

Function

Fit

Model

Eigenvalue

Classification

ability

All

Measures

-LRB-

-RRB-

PCA

-LRB-

-RRB-

As

before

we

split

the

data

set

into

programs

to

build

the

discriminant

function

and

the

remaining

binaries

to

verify

the

classification

ability

of

the

discriminant

function

We

perform

this

analysis

using

all

the

measures

and

the

principal

components

The

results

of

this

fit

and

classification

are

shown

below

in

table

Table

Discriminant

Analysis

For

Model

Fit

-LRB-

for

binaries

to

build

the

model

-RRB-

For

Test

Data

-LRB-

binaries

-RRB-

Model

Eigen

value

Classification

ability

Classification

ability

All

Measures

-LRB-

-RRB-

-LRB-

-RRB-

PCA

-LRB-

-RRB-

-LRB-

-RRB-

Table

shows

that

the

relative

code

churn

measures

have

effective

discriminant

ability

-LRB-

comparable

to

prior

studies

done

on

industrial

software

-LSB-

-RSB-

-RRB-

We

conclude

that

relative

code

churn

measures

can

be

used

to

discriminate

between

fault

and

not

fault-prone

binaries

-LRB-

H

-RRB-

Limitations

of

Study

Internal

validity

Internal

validity

issues

arise

when

there

are

errors

in

measurement

This

is

negated

to

an

extent

by

the

fact

that

the

entire

data

collection

process

is

automated

via

the

version

control

systems

However

the

version

control

systems

only

records

data

upon

developer

check-out

or

check-in

of

files

If

a

developer

made

many

overlapping

edits

to

a

file

in

a

single

check-out/check

in

period

then

a

certain

amount

of

churn

will

not

be

visible

A

developer

also

might

have

a

file

checked

out

for

a

very

long

period

of

time

during

which

few

changes

were

made

inflating

the

weeks

of

churn

measure

These

concerns

are

alleviated

to

some

extent

by

the

cross

check

among

the

measures

to

identify

abnormal

values

for

any

of

the

measures

and

the

huge

size

and

diversity

of

our

dataset

In

our

case

study

we

provide

evidence

for

using

all

the

relative

churn

measures

rather

than

a

subset

of

values

or

principal

components

This

is

case

study

specific

and

should

be

refined

based

on

further

results

External

validity

External

validity

issues

may

arise

from

the

fact

that

all

the

data

is

from

one

software

system

-LRB-

albeit

one

with

many

different

components

-RRB-

and

that

the

software

is

very

large

-LRB-

some

million

lines

of

code

-RRB-

as

other

software

systems

used

for

a

similar

analysis

might

not

be

of

comparable

size

We

have

shown

how

relative

code

churn

metrics

are

excellent

predictors

of

defect

density

in

a

large

industrial

software

system

Our

case

study

provides

strong

support

for

the

following

conclusions

Increase

in

relative

code

churn

measures

is

accompanied

by

an

increase

in

system

defect

density

Using

relative

values

of

code

churn

predictors

is

better

than

using

absolute

values

to

explain

the

system

defect

density

Relative

code

churn

measures

can

be

used

as

efficient

predictors

of

system

defect

density

and

Relative

code

churn

measures

can

be

used

to

discriminate

between

fault

and

not

fault-prone

binaries

We

plan

to

validate

our

approach

on

other

products

developed

inside

Microsoft

like

SQL

Server

and

Office

We

also

plan

to

develop

standards

for

all

the

measures

to

provide

guidance

to

the

developers

on

the

maximum

allowable

change

We

also

plan

to

investigate

how

testing

can

more

effectively

be

directed

towards

churned

code

We

would

like

to

express

our

appreciation

to

Brendan

Murphy

of

Microsoft

Research

for

providing

the

Windows

Server

data

set

We

would

like

to

thank

Madan

Musuvathi

of

Microsoft

Research

for

critical

feedback

on

the

relative

churn

measures

We

would

like

to

thank

Jim

Larus

of

Microsoft

Research

Laurie

Williams

Jason

Osborne

of

North

Carolina

State

University

for

reviewing

initial

drafts

of

this

paper

and

the

anonymous

referees

for

their

thoughtful

comments

on

an

earlier

draft

of

this

paper

-LSB-

-RSB-

ANSI/IEEE

IEEE

Standard

Glossary

of

Software

Engineering

Terminology

Standard

-LSB-

-RSB-

Basili

V.

Shull

F.

Lanubile

F.

Building

Knowledge

through

Families

of

Experiments

IEEE

Transactions

on

Software

Engineering

Vol

Vol

No.

No.

-LSB-

-RSB-

Boehm

B.

W.

Software

Engineering

Economics

Englewood

Cliffs

NJ

Prentice-Hall

Inc.

-LSB-

-RSB-

Brace

N.

Kemp

R.

Snelgar

R.

SPSS

for

Psychologists

Palgrave

Macmillan

-LSB-

-RSB-

Brito

e

Abreu

F.

Melo

W.

Evaluating

the

Impact

of

Object-Oriented

Design

on

Software

Quality

Proceedings

of

Third

International

Software

Metrics

Symposium

pp.

-LSB-

-RSB-

Denaro

G.

Pezze

M.

An

Empirical

Evaluation

of

Fault-Proneness

Models

Proceedings

of

International

Conference

on

Software

Engineering

pp.

-LSB-

-RSB-

Fenton

N.

E.

Ohlsson

N.

Quantitative

analysis

of

faults

and

failures

in

a

complex

software

system

IEEE

Transactions

on

Software

Engineering

Vol

No.

pp

-LSB-

-RSB-

Fenton

N.

E.

Pfleeger

S.L.

Software

Metrics

Boston

MA

International

Thompson

Publishing

-LSB-

-RSB-

Graves

T.

L.

Karr

A.F.

Marron

J.S.

Siy

H.

Predicting

Fault

Incidence

Using

Software

Change

History

IEEE

Transactions

on

Software

Engineering

Vol

No.

pp

-LSB-

-RSB-

Jackson

E.

J.

A

User

's

Guide

to

Principal

Components

John

Wiley

Sons

Inc.

-LSB-

-RSB-

Kaiser

H.

F.

An

Index

of

Factorial

Simplicity

Psychometrika

Vol

No.

pp.

-LSB-

-RSB-

Karunanithi

N.

A

Neural

Network

approach

for

Software

Reliability

Growth

Modeling

in

the

Presence

of

Code

Churn

Proceedings

of

International

Symposium

on

Software

Reliability

Engineering

pp.

-LSB-

-RSB-

Khoshgoftaar

T.

M.

Allen

E.B.

Goel

N.

Nandi

A.

McMullan

J.

Detection

of

Software

Modules

with

high

Debug

Code

Churn

in

a

very

large

Legacy

System

Proceedings

of

International

Symposium

on

Software

Reliability

Engineering

pp.

-LSB-

-RSB-

Khoshgoftaar

T.

M.

Allen

E.B.

Kalaichelvan

K.S.

Goel

N.

Hudepohl

J.P.

Mayrand

J.

Detection

of

fault-prone

program

modules

in

a

very

large

telecommunications

system

Proceedings

of

International

Symposium

Software

Reliability

Engineering

pp.

-LSB-

-RSB-

Khoshgoftaar

T.

M.

Szabo

R.M.

Improving

Code

Churn

Predictions

During

the

System

Test

and

Maintenance

Phases

Proceedings

of

IEEE

International

Conference

on

Software

Maintainence

pp.

-LSB-

-RSB-

Kleinbaum

D.

G.

Kupper

L.L.

Muller

K.E.

Applied

Regression

Analysis

and

Other

Multivariable

Methods

Boston

PWS-KENT

Publishing

Company

-LSB-

-RSB-

Munson

J.

C.

Elbaum

S.

Code

Churn

A

Measure

for

Estimating

the

Impact

of

Code

Change

Proceedings

of

IEEE

International

Conference

on

Software

Maintenance

pp.

-LSB-

-RSB-

Munson

J.

C.

Khoshgoftaar

T.M.

The

Detection

of

Fault-Prone

Programs

IEEE

Transactions

on

Software

Engineering

Vol

No.

pp.

-LSB-

-RSB-

Ohlsson

M.

C.

von

Mayrhauser

A.

McGuire

B.

Wohlin

C.

Code

Decay

Analysis

of

Legacy

Software

through

Successive

Releases

Proceedings

of

IEEE

Aerospace

Conference

pp.

-LSB-

-RSB-

Ostrand

T.

J.

Weyuker

E.J

Bell

R.M.

Where

the

Bugs

Are

Proceedings

of

the

ACM

SIGSOFT

International

Symposium

on

Software

Testing

and

Analysis

-LRB-

ISSTA

-RRB-

pp.

O O B I I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O B I I O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O B I I O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O B I I O O O O O O O O O B I O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

defect density
multiple regression
principal component analysis
relative code churn