On

the

Discovery

of

Significant

Statistical

Quantitative

Rules

In

this

paper

we

study

market

share

rules

rules

that

have

a

certain

market

share

statistic

associated

with

them

Such

rules

are

particularly

relevant

for

decision

making

from

a

business

perspective

Motivated

by

market

share

rules

in

this

paper

we

consider

statistical

quantitative

rules

-LRB-

SQ

rules

-RRB-

that

are

quantitative

rules

in

which

the

RHS

can

be

any

statistic

that

is

computed

for

the

segment

satisfying

the

LHS

of

the

rule

Building

on

prior

work

we

present

a

statistical

approach

for

learning

all

significant

SQ

rules

i.e.

SQ

rules

for

which

a

desired

statistic

lies

outside

a

confidence

interval

computed

for

this

rule

In

particular

we

show

how

resampling

techniques

can

be

effectively

used

to

learn

significant

rules

Since

our

method

considers

the

significance

of

a

large

number

of

rules

in

parallel

it

is

susceptible

to

learning

a

certain

number

of

false

rules

To

address

this

we

present

a

technique

that

can

determine

the

number

of

significant

SQ

rules

that

can

be

expected

by

chance

alone

and

suggest

that

this

number

can

be

used

to

determine

a

false

discovery

rate

for

the

learning

procedure

We

apply

our

methods

to

online

consumer

purchase

data

and

report

the

results

H.

-LSB-

Database

Management

-RSB-

Database

Applications

Data

Mining

Algorithms

Management

Rule

discovery

is

widely

used

in

data

mining

for

learning

interesting

patterns

Some

of

the

early

approaches

for

rule

learning

were

in

the

machine

learning

literature

-LSB-

-RSB-

More

recently

there

have

been

many

algorithms

-LSB-

-RSB-

proposed

in

the

data

mining

literature

most

of

which

are

based

on

the

concept

of

association

rules

-LSB-

-RSB-

While

all

these

various

approaches

have

been

successfully

used

in

many

applications

-LSB-

-RSB-

there

are

still

situations

that

these

types

of

rules

do

not

capture

The

problem

studied

in

this

paper

is

motivated

by

market

share

rules

a

specific

type

of

rule

that

can

not

be

represented

as

association

rules

Informally

a

market

share

rule

is

a

rule

that

specifies

the

market

share

of

a

product

or

a

firm

under

some

conditions

The

results

we

report

in

this

paper

are

from

real

user-level

Web

browsing

data

provided

to

us

by

comScore

Networks

The

data

consists

of

browsing

behavior

of

users

over

months

In

addition

to

customer

specific

attributes

two

attributes

in

a

transaction

that

are

used

to

compute

the

market

share

are

the

site

at

which

a

purchase

was

made

and

the

purchase

amount

Consider

the

example

rules

below

that

we

discovered

from

the

data

-LRB-

-RRB-

Household

Size

Income

ISP

Dialup

marketshare

Expedia

support

-LRB-

-RRB-

Region

North

East

Household

Size

marketshare

Expedia

support

-LRB-

-RRB-

Education

College

Region

West

Household

Eldest

Age

marketshare

Expedia

support

-LRB-

-RRB-

Household

Eldest

Age

marketshare

Expedia

support

The

market

share

for

a

specific

site

e.g.

Expedia.com

is

computed

as

the

dollar

value

of

flight

ticket

purchases

-LRB-

satisfying

the

LHS

of

the

rule

-RRB-

made

at

Expedia.com

divided

by

the

total

dollar

value

of

all

flight

ticket

purchases

satisfying

the

LHS

The

discovered

rules

suggest

that

Expedia

seems

to

be

doing

particularly

well

among

the

single

households

in

the

North

East

region

-LRB-

rule

-RRB-

while

it

cedes

some

market

in

the

segment

of

teenagers

-LRB-

rule

-RRB-

Rules

such

as

these

are

particularly

relevant

for

business

since

they

suggest

natural

actions

that

may

be

taken

For

example

it

may

be

worth

investigating

the

higher

market

share

segments

to

study

if

there

is

something

particularly

good

that

is

being

done

which

is

not

being

done

in

the

lower

market

share

segments

More

generally

market

share

is

an

example

of

a

statistic

that

is

computed

based

on

the

segment

satisfying

the

antecedent

of

the

rule

Besides

market

share

various

other

quantitative

statistics

on

the

set

of

transactions

satisfying

the

LHS

of

a

rule

can

be

computed

including

mean

and

variance

of

an

attribute

Prior

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

KDD

August

Seattle

Washington

USA

Copyright

ACM

Research

Track

Paper

work

on

learning

quantitative

association

rules

-LSB-

-RSB-

studied

the

discovery

of

rules

with

statistics

such

as

the

mean

variance

or

minimum/maximum

of

a

single

attribute

on

the

RHS

of

a

rule

In

this

paper

we

generalize

the

structure

of

the

rules

considered

in

-LSB-

-RSB-

to

rules

in

which

the

RHS

can

be

any

quantitative

statistic

that

can

be

computed

for

the

subset

of

data

satisfying

the

LHS

This

statistic

can

even

be

computed

based

on

multiple

attributes

We

term

such

rules

as

statistical

quantitative

rules

-LRB-

SQ

rules

-RRB-

With

respect

to

learning

SQ

rules

from

data

we

formulate

the

problem

as

learning

significant

SQ

rules

that

have

adequate

support

We

define

an

SQ

rule

to

be

significant

if

the

specific

statistic

computed

for

the

rule

lies

outside

a

certain

confidence

interval

This

confidence

interval

represents

a

range

in

which

the

statistic

can

be

expected

by

chance

alone

This

is

an

important

range

to

identify

if

the

rules

discovered

are

to

be

interpreted

as

suggesting

fundamental

relationships

between

the

LHS

and

the

market

share

For

example

by

chance

alone

if

it

is

highly

likely

that

the

market

share

of

Expedia

is

between

and

for

any

subset

of

data

then

it

is

not

at

all

clear

that

the

rule

relating

income

and

Expedia

's

market

share

-LRB-

rule

in

the

example

-RRB-

is

identifying

a

fundamental

relationship

between

income

and

the

market

share

While

prior

work

-LSB-

-RSB-

has

used

confidence

intervals

to

identify

significant

rules

most

of

these

approaches

are

either

parametric

or

specific

for

binary

data

Building

on

prior

work

in

this

paper

we

present

a

statistical

approach

for

learning

significant

SQ

rules

that

is

entirely

non-parametric

In

particular

we

show

how

resampling

techniques

such

as

permutation

can

be

effectively

used

to

learn

confidence

intervals

for

rules

Based

on

these

confidence

intervals

significant

rules

can

be

identified

However

since

our

method

considers

the

significance

of

a

large

number

of

rules

in

parallel

for

a

given

significance

level

it

is

susceptible

to

learning

a

certain

number

of

false

rules

To

address

this

we

present

an

intuitive

resampling

technique

that

can

determine

the

number

of

false

rules

and

argue

that

this

number

can

be

used

to

determine

a

false

discovery

rate

for

the

learning

procedure

The

practical

significance

of

this

approach

is

that

we

learn

significant

SQ

rules

from

data

and

specify

what

the

false

discovery

rate

exactly

is

The

paper

is

organized

as

follows

We

first

define

SQ

rules

in

the

next

section

Section

presents

an

algorithm

for

computing

confidence

intervals

and

Section

presents

an

algorithm

for

learning

significant

SQ

rules

In

Section

we

explain

how

the

false

discovery

rate

for

our

approach

can

be

computed

We

present

detailed

experimental

results

on

real

web

browsing

data

in

Section

followed

by

a

literature

review

and

conclusions

In

this

section

we

define

SQ

rules

and

significant

SQ

rules

Let

A

-LCB-

A

A

A

n

-RCB-

be

a

set

of

attributes

that

will

be

used

to

describe

segments

and

B

-LCB-

B

B

B

m

-RCB-

be

another

set

of

attributes

that

will

be

used

to

compute

various

statistics

that

describe

the

segment

Let

dom

-LRB-

A

i

-RRB-

and

dom

-LRB-

B

j

-RRB-

represent

the

set

of

values

that

can

be

taken

by

attribute

A

i

and

B

j

respectively

for

any

A

i

A

and

B

j

B.

Let

D

be

a

dataset

of

N

transactions

where

each

transaction

is

of

the

form

-LCB-

A

a

A

a

A

n

a

n

B

b

B

b

B

m

b

m

-RCB-

where

a

i

dom

-LRB-

A

i

-RRB-

and

b

j

dom

-LRB-

B

j

-RRB-

Let

an

atomic

condition

be

a

proposition

of

the

form

value

A

i

value

for

ordered

attributes

and

A

i

value

for

unordered

attributes

where

value

value

value

belong

to

the

finite

set

of

discrete

values

taken

by

A

i

in

D.

Finally

let

an

itemset

represent

a

conjunction

of

atomic

conditions

Definition

-LRB-

SQ

rule

-RRB-

Given

-LRB-

i

-RRB-

sets

of

attributes

A

and

B

-LRB-

ii

-RRB-

a

dataset

D

and

-LRB-

iii

-RRB-

a

function

f

that

computes

a

desired

statistic

of

interest

on

any

subset

of

data

an

SQ

rule

is

a

rule

of

the

form

X

f

-LRB-

D

X

-RRB-

statistic

support

sup

-LRB-

-RRB-

where

X

is

an

itemset

involving

attributes

in

A

only

D

X

is

the

subset

of

D

satisfying

X

the

function

f

computes

some

statistic

from

the

values

of

the

B

attributes

in

the

subset

D

X

and

support

is

the

number

of

transactions

in

D

satisfying

X.

Note

that

the

statistic

on

the

RHS

of

the

rule

can

be

computed

using

the

values

of

multiple

attributes

The

following

examples

are

listed

to

demonstrate

different

types

of

rules

that

an

SQ

rule

can

represent

For

ease

of

exposition

we

use

the

name

of

the

desired

statistic

in

the

RHS

instead

of

referring

to

it

as

f

-LRB-

D

X

-RRB-

Quantitative

association

rules

-LSB-

-RSB-

population-subset

mean

or

variance

values

for

the

subset

-LRB-

-RRB-

Quantitative

association

rules

are

a

popular

representation

for

rules

in

the

data

mining

literature

in

which

the

RHS

of

a

rule

represents

the

mean

or

variance

of

some

attribute

Example

Education

graduate

Mean

-LRB-

purchase

-RRB-

-LRB-

-RRB-

is

a

special

case

of

-LRB-

-RRB-

where

f

-LRB-

subset

-RRB-

is

the

mean

of

some

attribute

B

j

in

the

subset

of

data

Market

share

rules

Let

-LCB-

A

A

A

n

MSV

P

-RCB-

be

a

set

of

attributes

in

a

dataset

D.

MSV

-LRB-

Market

Share

Variable

-RRB-

is

a

special

categorical

attribute

for

which

the

market

share

values

are

to

be

computed

P

is

a

special

continuous

variable

that

is

the

basis

for

the

market

share

computation

for

MSV

For

example

each

transaction

T

k

may

represent

a

book

purchased

online

A

through

A

n

may

represent

attributes

of

the

customer

who

makes

the

purchase

such

as

income

region

of

residence

and

household

size

For

each

transaction

MSV

is

the

variable

indicating

the

online

book

retailer

where

the

purchase

was

made

dom

-LRB-

MSV

-RRB-

may

be

-LCB-

Amazon

Barnes

Noble

Ebay

-RCB-

and

P

is

the

price

of

the

book

purchased

For

a

specific

v

dom

-LRB-

MSV

-RRB-

a

market

share

statistic

can

be

computed

as

described

below

Market

share

rules

have

the

following

form

X

marketshare

v

msh

support

sup

-LRB-

-RRB-

where

X

is

an

itemset

consisting

of

attributes

in

-LCB-

A

A

A

n

-RCB-

and

marketshare

v

is

a

statistic

that

represents

the

market

share

of

a

specific

v

dom

-LRB-

MSV

-RRB-

This

is

computed

as

follows

Let

D

X

represent

the

subset

of

transactions

satisfying

X

and

D

X

MSV

v

In

association

rules

support

is

the

number

of

transactions

satisfying

both

LHS

and

RHS

of

a

rule

In

SQ

rules

since

the

RHS

is

not

an

itemset

we

define

support

as

the

number

of

transactions

satisfying

the

LHS

of

a

rule

only

The

provider

comScore

Networks

categorizes

each

purchase

into

categories

such

as

book

travel

and

consumer

electronics

Hence

we

can

generate

datasets

in

which

all

transactions

represent

purchases

in

a

single

category

and

this

helps

in

the

generation

of

market

share

rules

representing

specific

categories

Research

Track

Paper

represent

the

subset

of

transactions

satisfying

-LRB-

X

MSV

v

-RRB-

Then

marketshare

v

is

computed

as

sum

-LRB-

P

D

X

MSV

v

-RRB-

sum

-LRB-

P

D

X

-RRB-

where

sum

-LRB-

P

D

-RRB-

is

the

sum

of

all

the

values

of

attribute

P

in

the

transactions

in

D.

Market

share

rules

naturally

occur

in

various

applications

including

online

purchases

at

various

Web

sites

sales

applications

and

knowledge

management

applications

The

examples

presented

in

the

introduction

are

real

market

share

rules

discovered

in

online

purchase

data

The

following

additional

examples

illustrate

the

versatility

and

usefulness

of

market

share

rules

Within

a

specific

product

category

-LRB-

e.g.

shoes

-RRB-

Footlocker

sells

competing

brands

of

shoes

In

their

transaction

data

the

brand

of

the

shoe

can

be

the

MSV

and

the

purchase

price

is

P.

Consider

a

dataset

of

patents

associated

with

some

area

-LRB-

e.g.

hard

disks

-RRB-

Each

record

may

consist

of

several

attributes

describing

a

patent

including

one

attribute

-LRB-

MSV

-RRB-

which

represents

the

organization

to

which

the

patent

belongs

and

another

attribute

that

is

always

-LRB-

representing

P

and

indicating

a

granted

patent

-RRB-

in

the

data

For

a

specific

organization

e.g.

IBM

market

share

rules

will

represent

the

percentage

of

patents

that

belong

to

IBM

under

some

conditions

involving

other

attributes

of

the

patent

Definition

differs

from

the

definition

of

quantitative

rule

-LSB-

-RSB-

as

follows

First

it

is

not

limited

to

mean

and

variance

statistics

and

assumes

a

much

broader

class

of

statistics

including

the

market

share

statistics

Second

unlike

quantitative

rules

the

statistic

of

interest

in

the

RHS

of

a

rule

can

be

computed

based

on

multiple

attributes

Definition

-LRB-

Significant

SQ

rule

-RRB-

For

a

given

significance

level

-LRB-

-RRB-

let

-LRB-

stat

L

stat

H

-RRB-

be

the

-LRB-

-RRB-

confidence

interval

for

a

desired

statistic

where

this

confidence

interval

represents

the

range

in

which

the

statistic

can

be

expected

by

chance

alone

An

SQ

rule

X

f

-LRB-

D

X

-RRB-

statistic

support

sup

is

significant

if

statistic

lies

outside

the

range

-LRB-

stat

L

stat

H

-RRB-

The

main

objective

of

this

paper

is

to

discover

all

significant

SQ

rules

The

first

challenge

in

learning

significant

SQ

rules

is

in

constructing

a

confidence

interval

for

the

desired

statistic

such

that

this

interval

represents

a

range

of

values

for

the

RHS

statistic

that

can

be

expected

by

chance

alone

In

the

next

section

we

present

an

algorithm

for

learning

these

confidence

intervals

The

first

question

that

needs

to

be

addressed

is

what

is

meant

by

a

range

for

the

statistic

that

can

be

expected

by

chance

alone

In

this

section

we

start

by

addressing

this

question

and

outline

a

procedure

by

which

such

a

range

can

be

computed

Next

we

will

point

out

the

computational

challenge

in

implementing

such

a

procedure

for

learning

these

intervals

for

several

SQ

rules

and

then

outline

three

observations

that

will

substantially

help

address

the

computational

problems

Based

on

these

observations

we

present

a

resampling-based

algorithm

for

computing

the

confidence

intervals

Motivation

and

outline

of

a

procedure

For

a

given

SQ

rule

the

desired

confidence

interval

theoretically

represents

the

range

in

which

the

statistic

can

be

expected

when

there

is

no

fundamental

relationship

between

the

LHS

of

the

rule

and

the

statistic

More

precisely

since

the

statistic

is

computed

from

the

values

of

the

B

attributes

the

confidence

interval

represents

the

range

in

which

the

statistic

can

be

expected

when

the

A

attributes

are

truly

independent

of

the

B

attributes

Without

making

any

parametric

distributional

assumptions

such

a

confidence

interval

can

be

generated

using

the

classical

nonparametric

technique

of

permutation

Indeed

permutation-based

approaches

have

been

commonly

used

to

generate

confidence

intervals

in

the

statistics

literature

-LSB-

-RSB-

If

R

is

the

set

of

all

attributes

in

a

dataset

the

basic

idea

in

permutation

is

to

create

multiple

datasets

by

randomly

permuting

the

values

of

some

attributes

R

i

R

Such

a

permutation

would

create

a

dataset

in

which

R

i

is

independent

of

-LRB-

R

R

i

-RRB-

but

would

maintain

the

distributions

of

R

i

and

-LRB-

R

R

i

-RRB-

in

the

permutation

dataset

to

be

the

same

as

the

distributions

of

these

attributes

in

the

original

dataset

Table

illustrates

one

example

of

a

permutation

dataset

D

in

which

the

B

attributes

are

randomly

permuted

Since

a

desired

statistic

can

be

computed

on

each

permutation

dataset

a

distribution

for

the

statistic

can

be

computed

based

on

its

values

from

the

multiple

permutation

datasets

A

confidence

interval

can

then

be

computed

from

this

distribution

Table

Dataset

permutation

Original

dataset

D

Permutation

dataset

D

A

A

B

B

A

A

B

B

As

mentioned

above

this

is

a

commonly

used

procedure

in

nonparametric

statistics

The

reason

this

procedure

makes

sense

is

as

follows

Even

if

there

is

a

relationship

between

the

LHS

of

an

SQ

rule

and

the

statistic

on

the

RHS

by

holding

the

A

attributes

fixed

and

randomly

re-ordering

the

values

of

the

B

attributes

the

relationship

is

destroyed

and

the

A

attributes

and

B

attributes

are

now

independent

of

each

other

Repeating

this

procedure

many

times

provides

many

datasets

in

which

the

A

attributes

and

B

attributes

are

independent

of

each

other

while

maintaining

the

distributions

of

the

A

and

B

attributes

to

be

the

same

as

their

distributions

in

the

original

dataset

The

values

for

the

statistic

computed

from

the

many

permutation

datasets

is

used

to

construct

a

distribution

for

the

statistic

that

can

be

expected

when

the

A

attributes

are

truly

independent

of

the

B

attributes

Specifically

for

the

same

itemset

X

compare

the

following

two

SQ

rules

in

D

and

D

D

X

f

-LRB-

X

D

-RRB-

stat

D

support

sup

D

-LRB-

-RRB-

D

X

f

-LRB-

X

D

-RRB-

stat

D

support

sup

D

-LRB-

-RRB-

First

note

that

the

supports

of

the

rules

are

the

same

since

the

number

of

records

satisfying

X

in

the

permutation

dataset

is

the

same

as

the

original

dataset

We

will

use

this

observation

to

build

a

more

efficient

method

for

computing

confidence

intervals

Research

Track

Paper

shortly

A

confidence

interval

for

the

rule

in

-LRB-

-RRB-

can

be

computed

using

the

following

ve

procedure

Create

permutation

dataset

D

from

the

original

dataset

D

and

compute

stat

D

-LRB-

as

mentioned

earlier

in

Section

the

function

f

computes

this

number

based

on

the

records

satisfying

X

-RRB-

Repeat

step

N

perm

times

sort

all

the

N

perm

stat

D

values

in

an

ascending

order

-LRB-

stat

D

stat

D

stat

D

Nperm

-RRB-

and

let

the

th

and

-LRB-

-RRB-

th

percentiles

from

this

list

be

stat

D

L

and

stat

D

H

The

N

perm

values

computed

above

represents

a

distribution

for

the

statistic

that

can

be

expected

by

chance

alone

while

the

percentile

values

from

this

distribution

determine

a

specific

confidence

interval

-LRB-

Below

we

use

the

terms

distribution

and

confidence

interval

frequently

-RRB-

The

-LRB-

-RRB-

confidence

interval

for

the

SQ

rule

in

Equation

-LRB-

-RRB-

is

-LRB-

stat

D

L

stat

D

H

-RRB-

Computational

challenges

and

solutions

Computing

these

confidence

intervals

for

multiple

candidate

SQ

rules

creates

several

computational

problems

which

we

will

address

in

this

section

For

example

if

we

need

to

test

potential

significant

rules

-LRB-

which

is

a

reasonably

small

number

for

data

mining

tasks

-RRB-

then

we

would

need

to

repeat

the

above

steps

times

and

this

means

generating

permutation

datasets

N

perm

times

and

to

compute

the

desired

statistic

in

each

permutation

dataset

The

following

observations

substantially

reduce

the

computational

complexity

of

the

procedure

Sampling

can

be

used

instead

of

creating

permutation

datasets

For

the

SQ

rule

in

Equation

-LRB-

-RRB-

computing

stat

D

on

a

permutation

dataset

is

really

equivalent

to

computing

stat

D

based

on

a

random

sample

of

sup

D

records

in

D

This

is

the

case

since

of

the

A

attributes

play

a

role

in

the

computation

of

the

statistic

Permuting

the

dataset

identifying

the

-LRB-

sup

D

-RRB-

records

where

X

holds

and

then

computing

the

statistic

on

this

subset

achieves

the

same

effect

as

picking

a

random

sample

of

sup

D

records

from

D

and

computing

the

statistic

on

this

random

subset

Hence

to

determine

the

confidence

interval

for

the

SQ

rule

in

Equation

-LRB-

-RRB-

instead

of

permuting

the

dataset

N

perm

times

it

is

enough

to

sample

sup

D

records

from

D

for

N

perm

times

Some

of

the

candidate

SQ

rules

have

the

same

support

values

as

other

rules

Based

on

this

observation

confidence

intervals

for

two

SQ

rules

with

the

same

support

can

be

approximated

by

the

same

interval

This

is

the

case

since

for

a

given

rule

the

interval

is

generated

by

sampling

sup

D

records

many

times

and

if

another

rule

has

support

sup

D

then

the

interval

for

that

rule

will

be

similar

if

the

same

procedure

is

repeated

-LRB-

it

will

not

be

exactly

the

N

perm

is

typically

a

big

number

If

we

let

N

perm

N

which

is

the

number

of

all

possible

permutations

we

will

be

implementing

a

Monte

Carlo

test

On

large

datasets

such

a

test

is

impractical

For

a

statistic

like

market

share

whose

value

is

limited

by

and

N

perm

makes

the

distribution

precise

to

the

third

decimal

place

In

our

experiments

N

perm

Since

we

do

not

have

any

prior

assumption

about

the

expected

value

of

the

statistic

we

use

a

two

sided

p-value

same

because

of

randomization

-RRB-

Therefore

fewer

confidence

intervals

need

to

be

generated

It

is

adequate

to

generate

a

fixed

number

of

intervals

independent

of

the

number

of

rules

considered

We

observe

that

the

interval

for

an

SQ

rule

with

support

sup

D

can

be

approximated

by

an

interval

computed

by

sampling

sup

E

records

where

sup

E

is

reasonably

close

to

sup

D

This

is

a

heuristic

that

we

use

to

considerably

reduce

the

complexity

of

the

procedure

Denote

N

Rule

as

the

number

of

rules

to

be

tested

If

all

rules

have

different

support

values

we

need

to

construct

N

Rule

distributions

Instead

we

would

construct

a

fixed

number

N

dist

distributions

such

that

for

rule

X

f

-LRB-

D

X

-RRB-

statistic

support

sup

statistic

is

compared

with

the

distribution

that

is

constructed

by

sampling

the

closest

number

of

transactions

to

sup

This

heuristic

is

more

meaningful

when

we

consider

support

in

terms

of

percentage

of

transactions

satisfying

LHS

of

a

rule

which

is

a

number

between

and

Algorithm

CIComp

Based

on

the

above

observations

we

present

in

Figure

algorithm

CIComp

for

constructing

N

dist

distributions

and

determining

the

-LRB-

-RRB-

confidence

intervals

for

a

given

significance

level

Input

dataset

D

with

N

transactions

the

number

of

distributions

N

dist

the

number

of

points

in

each

distribution

N

perm

a

function

f

that

computes

the

desired

statistic

and

significance

level

Output

N

dist

distributions

and

significance

thresholds

for

-LRB-

dist

dist

N

dist

dist

-RRB-

-LCB-

N

sample

dist

N

dist

N

for

-LRB-

perm

perm

N

perm

perm

-RRB-

-LCB-

S

N

sample

transactions

from

D

sampled

without

replacements

stat

-LSB-

dist

-RSB-

-LSB-

perm

-RSB-

f

-LRB-

S

-RRB-

-RCB-

sort

-LRB-

stat

-LSB-

dist

-RSB-

-RRB-

LowerCI

-LSB-

dist

-RSB-

stat

-LSB-

dist

-RSB-

-LSB-

-LRB-

N

perm

-RRB-

-RSB-

UpperCI

-LSB-

dist

-RSB-

stat

-LSB-

dist

-RSB-

-LSB-

-LRB-

N

perm

-RRB-

-LRB-

-RRB-

-RSB-

-RCB-

Output

stat

-LSB-

-RSB-

-LSB-

-RSB-

LowerCI

-LSB-

-RSB-

UpperCI

-LSB-

-RSB-

Figure

Algorithm

CIComp

In

the

above

algorithm

N

dist

N

perm

and

are

user-defined

parameters

is

usually

chosen

to

be

or

For

N

dist

and

N

perm

the

larger

they

are

the

more

precise

the

distributions

will

be

Let

N

N

dist

N

perm

and

We

use

these

numbers

as

an

example

to

explain

the

algorithm

For

step

the

first

distribution

corresponds

to

N

sample

dist/N

dist

N

transactions

Step

to

computes

N

perm

statistics

for

randomly

sampled

transactions

from

dataset

D

Then

we

sort

these

statistics

and

pick

and

percentiles

which

are

the

th

and

th

numbers

in

the

distribution

as

the

lower

and

upper

thresholds

for

the

-LRB-

-RRB-

confidence

interval

Steps

through

are

repeated

N

dist

times

to

get

the

desired

number

of

distributions

and

confidence

intervals

If

the

sampling

is

done

with

replacement

then

the

interval

will

be

the

bootstrap

confidence

interval

The

two

intervals

will

essentially

be

the

same

when

the

support

of

the

itemset

is

small

Research

Track

Paper

The

computation

complexity

of

the

algorithm

in

Figure

is

O

-LRB-

N

N

perm

N

dist

-RRB-

whereas

the

complexity

of

ve

method

is

O

-LRB-

N

N

perm

N

rule

-RRB-

Note

that

N

dist

can

be

fixed

to

a

reasonable

small

number

e.g.

whereas

N

rule

is

the

number

of

rules

that

are

being

tested

and

can

easily

be

orders

of

magnitude

more

than

N

dist

Given

the

distributions

and

confidence

intervals

discovering

all

significant

statistical

rules

is

straightforward

Algorithm

SigSQrules

is

presented

in

Figure

Input

dataset

D

with

N

transactions

sets

of

attributes

A

and

B

N

dist

stat

-LSB-

-RSB-

-LSB-

-RSB-

LowerCI

-LSB-

-RSB-

and

UpperCI

-LSB-

-RSB-

from

algorithm

CIComp

a

function

f

that

computes

the

desired

statistic

minimum

support

minsup

and

a

large

itemset

generation

procedure

largeitemsets

Output

set

of

Significant

rules

sigrules

L

largeitemsets

-LRB-

D

A

minsup

-RRB-

generates

large

itemsets

involving

attributes

in

A

sigrules

-LCB-

-RCB-

forall

-LRB-

itemsets

x

L

-RRB-

-LCB-

x.stat

f

-LRB-

D

x

-RRB-

statistic

computed

on

transactions

satisfying

x

dist

round

-LRB-

support

-LRB-

x

-RRB-

N

N

dist

-RRB-

if

x

stat

-LRB-

LowerCI

-LSB-

dist

-RSB-

UpperCI

-LSB-

dist

-RSB-

-RRB-

-LCB-

x

f

-LRB-

D

x

-RRB-

x

stat

is

significant

x.pvalue

percentile

of

x.stat

in

stat

-LSB-

dist

-RSB-

-LSB-

N

perm

-RSB-

sigrules

sigrules

-LCB-

x

f

-LRB-

D

x

-RRB-

x

stat

support

support

-LRB-

x

-RRB-

-RCB-

-RCB-

-RCB-

Figure

Algorithm

SigSQrules

Given

N

dist

distributions

constructed

from

the

algorithm

CIComp

we

use

the

above

algorithm

to

discover

all

significant

SQ

rules

We

continue

to

use

the

example

N

N

dist

and

N

perm

to

describe

the

steps

in

Figure

Note

that

the

attributes

in

A

represent

the

attributes

in

the

dataset

that

are

used

to

describe

segments

for

which

statistics

can

be

computed

Step

uses

any

large

itemset

generation

procedure

in

rule

discovery

literature

to

generate

all

large

itemsets

involving

attributes

in

A

The

exact

procedure

used

will

depend

on

whether

the

attributes

in

A

are

all

categorical

or

not

If

they

are

then

Apriori

algorithm

can

be

used

to

learn

all

large

itemsets

If

some

of

them

are

continuous

then

other

methods

such

as

the

ones

described

in

-LSB-

-RSB-

can

be

used

Step

computes

the

statistic

function

for

each

large

itemset

x

In

step

we

find

out

which

distribution

is

to

be

used

for

significance

test

For

example

if

support

-LRB-

x

-RRB-

then

support

-LRB-

x

-RRB-

N

N

dist

-LRB-

-RRB-

and

hence

dist

will

be

round

-LRB-

-RRB-

We

would

compare

x.stat

with

its

corresponding

confidence

interval

-LRB-

LowerCI

-LSB-

-RSB-

UpperCI

-LSB-

-RSB-

-RRB-

in

step

If

x.stat

is

outside

of

the

confidence

interval

the

rule

is

significant

and

we

use

step

to

calculate

its

p-value

If

x.stat

is

the

qth

percentile

the

p-value

is

min

-LRB-

q

q

-RRB-

The

p-value

is

not

only

a

value

to

understand

how

significant

a

rule

is

but

is

also

useful

for

determining

the

false

discovery

rate

in

Section

Note

that

the

confidence

interval

used

to

test

significance

of

a

rule

is

approximate

since

we

do

not

compute

this

interval

for

the

exact

value

of

the

support

of

this

rule

Instead

we

use

the

closest

interval

-LRB-

which

was

pre-computed

as

described

in

Section

-RRB-

corresponding

to

this

support

value

In

future

research

we

will

quantify

the

effects

of

this

approximation

We

would

also

like

to

point

out

that

in

many

cases

-LRB-

see

below

-RRB-

the

computation

of

the

statistic

can

be

done

efficiently

within

the

itemset

generation

procedure

-LRB-

largeitems

-RRB-

itself

This

can

be

used

to

modify

the

algorithm

to

make

it

more

efficient

once

a

specific

itemset

generation

procedure

is

used

This

is

the

case

if

the

function

f

that

computes

the

statistic

on

transactions

T

T

T

s

is

a

recursive

function

on

s

that

is

f

-LRB-

T

T

T

s

-RRB-

g

-LRB-

f

-LRB-

T

T

T

-RRB-

f

-LRB-

T

s

-RRB-

s

-RRB-

-LRB-

-RRB-

Many

statistics

such

as

mean

and

market

share

are

recursive

For

example

Mean

-LRB-

T

T

T

s

-RRB-

-LSB-

Mean

-LRB-

T

T

T

s

-RRB-

-LRB-

s

-RRB-

Mean

-LRB-

T

s

-RRB-

-RSB-

s

In

this

section

we

presented

an

algorithm

SigSQrules

for

generating

significant

SQ

rules

However

as

mentioned

in

the

introduction

for

any

given

level

of

significance

for

a

rule

the

fact

that

thousands

of

rules

are

evaluated

for

their

significance

makes

it

possible

to

discover

a

certain

number

of

false

rules

This

is

the

well

known

multiple

hypothesis

testing

problem

-LSB-

-RSB-

While

it

is

difficult

to

eliminate

this

problem

it

is

possible

to

quantify

this

effect

In

the

next

section

we

discuss

the

problem

of

false

discovery

in

detail

and

present

an

algorithm

for

determining

the

false

discovery

rate

associated

with

the

discovery

of

significant

SQ

rules

As

mentioned

above

when

multiple

rules

are

tested

in

parallel

for

significance

it

is

possible

to

learn

a

number

of

false

rules

by

chance

alone

Indeed

this

is

a

problem

for

many

rule

discovery

methods

in

the

data

mining

literature

The

false

discovery

rate

-LRB-

FDR

-RRB-

is

the

expected

percentage

of

false

rules

among

all

the

discovered

rules

Prior

work

in

statistics

has

taken

two

approaches

to

deal

with

the

multiple

hypothesis

testing

problem

-LSB-

-RSB-

One

approach

attempts

to

lower

the

false

discovery

rate

by

adjusting

the

significance

level

at

which

each

rule

is

tested

As

we

will

describe

below

this

approach

is

not

suitable

for

data

mining

since

it

will

result

in

very

few

rules

being

discovered

The

second

approach

assumes

that

a

given

number

of

false

discoveries

should

be

expected

and

focuses

on

estimating

what

the

false

discovery

rate

-LRB-

FDR

-RRB-

exactly

is

This

is

more

useful

for

data

mining

since

it

permits

the

discovery

of

a

reasonable

number

of

rules

but

at

the

same

time

computes

a

FDR

that

can

give

users

an

idea

of

what

percentage

of

the

discovered

rules

are

spurious

In

this

section

we

first

review

key

ideas

related

to

the

multiple

hypotheses

testing

problem

and

then

present

a

nonparametric

method

to

determine

false

discovery

rate

for

our

procedure

For

significance

tests

for

a

single

rule

the

significance

level

is

defined

as

the

probability

of

discovering

a

significant

rule

when

the

LHS

and

RHS

of

the

rule

are

actually

independent

of

each

other

in

other

words

is

the

probability

of

a

false

-LRB-

spurious

-RRB-

discovery

For

example

on

a

random

dataset

where

all

attributes

are

independent

if

we

test

rules

then

by

definition

of

we

expect

false

discoveries

by

pure

chance

alone

When

some

of

the

attributes

are

dependent

on

each

other

as

is

the

case

for

most

datasets

on

which

rule

discovery

methods

are

used

the

above

approach

can

not

be

used

to

get

an

expectation

for

the

number

of

false

rules

In

such

cases

two

approaches

are

Research

Track

Paper

possible

In

statistics

a

measure

called

familywise

error

rate

-LRB-

FWER

-RRB-

is

defined

as

the

probability

of

getting

at

least

one

false

rule

output

Most

conventional

approaches

in

statistics

that

deals

with

the

multiple

hypotheses

testing

problem

use

different

methods

to

control

FWER

by

lowering

significance

level

for

individual

rule

ind

For

example

Bonferroni-type

procedures

would

have

ind

the

number

of

rules

tested

which

is

However

when

the

number

of

hypotheses

tested

is

large

-LRB-

as

is

the

case

in

data

mining

algorithms

-RRB-

extreme

low

value

e.g.

will

result

in

very

few

rules

discovered

The

other

type

of

approach

as

taken

recently

in

-LSB-

-RSB-

estimates

the

false

discovery

rate

-LRB-

FDR

-RRB-

the

expectation

of

the

proportion

of

false

discoveries

in

all

discoveries

Table

Confusion

matrix

of

the

number

of

rules

Non-Significant

Rules

Significant

Rules

LHS

independent

of

RHS

a

b

LHS

dependent

on

RHS

c

d

In

Table

the

number

of

rules

tested

is

-LRB-

a

b

c

d

-RRB-

out

of

which

-LRB-

a

b

-RRB-

is

the

number

of

rules

where

the

LHS

of

the

rules

is

truly

independent

of

the

RHS

and

-LRB-

c

d

-RRB-

is

the

number

of

rules

where

there

is

a

real

relationship

between

the

LHS

and

the

RHS

of

the

rules

The

columns

determine

how

many

tested

rules

are

output

as

significant

or

non-significant

The

two

terms

FDR

and

FWER

can

be

defined

precisely

as

FDR

Exp

-LRB-

b

b

d

-RRB-

and

FWER

Prob

-LRB-

b

-RRB-

We

adopt

FDR

estimation

in

this

section

because

it

effectively

estimates

false

discoveries

without

rejecting

too

many

discovered

rules

However

the

method

proposed

in

the

literature

-LSB-

-RSB-

for

FDR

can

not

be

used

for

large

scale

rule

discovery

because

of

the

following

two

reasons

first

the

assumption

that

statistics

of

the

rules

tested

are

independent

from

each

other

-LRB-

which

some

of

the

approaches

use

-RRB-

is

not

true

For

example

rules

A

Mean

-LRB-

D

-RRB-

and

A

A

Mean

-LRB-

D

-RRB-

are

not

independent

In

fact

a

large

number

of

rules

are

related

to

each

other

in

rule

discovery

because

their

LHS

share

common

conditions

and

RHS

come

from

the

same

attributes

Second

methods

in

statistics

draw

conclusions

based

on

the

number

of

rules

tested

-LRB-

a

b

c

d

-RRB-

however

as

indicated

in

-LSB-

-RSB-

a

and

c

are

unknown

values

due

to

the

filtering

by

support

constraint

Without

making

any

assumptions

below

we

present

another

permutation-based

method

to

estimate

the

FDR

for

our

procedure

for

learning

significant

SQ

rules

Denote

N

sig

-LRB-

-RRB-

to

be

the

number

of

significant

rules

discovered

from

dataset

D

when

the

significant

level

In

Table

N

sig

-LRB-

-RRB-

b

d.

Similar

to

the

procedure

described

in

Section

by

keeping

the

values

in

attributes

A

intact

and

permuting

the

B

attributes

we

get

a

permutation

dataset

D

Since

we

remove

any

relationship

between

A

and

B

attributes

by

this

procedure

all

the

LHS

and

RHS

statistic

of

each

rule

tested

in

D

are

independent

If

we

apply

the

significant

rule

discovery

algorithm

SigSQrules

the

number

of

significant

rules

discovered

from

D

when

the

significant

level

will

be

one

instance

of

false

discovery

that

is

N

sig-perm

-LRB-

-RRB-

b

It

is

easy

to

see

that

by

creating

multiple

permutation

datasets

we

can

estimate

the

expectation

of

the

number

of

false

discoveries

and

thus

compute

a

false

discovery

rate

FDR

Exp

-LRB-

N

sig-perm

-LRB-

-RRB-

-RRB-

N

sig

-LRB-

-RRB-

We

will

describe

the

steps

how

FDR

-LRB-

-RRB-

can

be

estimated

in

detail

in

the

Appendix

In

this

section

we

described

the

problem

of

multiple

hypotheses

testing

and

pointed

out

that

for

any

given

significance

level

a

certain

number

of

significant

SQ

rules

will

be

discovered

by

chance

alone

We

then

described

an

intuitive

permutation

based

procedure

to

compute

the

false

discovery

rate

From

a

practical

point

of

view

the

procedure

described

above

can

be

used

in

conjunction

with

SigSQrules

to

discover

a

set

of

significant

SQ

rules

and

provide

a

number

representing

the

percentage

of

these

rules

that

are

likely

to

be

spurious

In

this

section

we

present

results

from

learning

significant

market

share

rules

a

specific

type

of

SQ

rules

We

started

with

user-level

online

purchase

data

gathered

by

comScore

Networks

a

market

data

vendor

The

data

consist

of

users

online

browsing

and

purchase

behavior

over

a

period

of

six

months

The

market

data

vendor

tracks

all

online

purchases

explicitly

by

parsing

the

content

of

all

pages

delivered

to

each

user

Starting

from

the

raw

data

we

created

a

dataset

of

purchases

where

each

transaction

represents

a

purchase

made

at

an

online

retailer

Attributes

of

the

transaction

include

user

demographics

the

site

at

which

the

purchase

was

made

the

primary

category

-LRB-

e.g.

books

home

electronics

etc

-RRB-

of

the

site

the

product

purchased

and

the

price

paid

for

the

product

Within

a

specific

category

e.g.

books

significant

market

share

rules

would

be

particularly

interesting

to

discover

We

selected

many

datasets

with

purchases

belonging

to

each

specific

category

and

applied

our

method

to

learn

several

interesting

significant

market

share

rules

For

space

limitations

we

do

not

present

all

the

results

but

report

the

results

for

learning

market

share

rules

for

the

top

three

retailers

in

the

online

book

industry

Specifically

the

dataset

consists

of

all

transactions

in

which

a

book

was

purchased

at

any

site

and

we

use

the

methods

presented

in

the

paper

to

learn

market

share

rules

for

the

top

sites

Amazon.com

Barnes

Noble

and

Ebay

The

total

number

of

transactions

was

records

and

we

limit

the

rules

to

having

at

most

five

items

on

the

LHS

of

a

rule

Rule

Examples

Among

the

most

significant

market

share

rules

-LRB-

as

determined

by

the

p-values

of

these

rules

-RRB-

we

picked

four

rules

to

list

that

were

particularly

interesting

for

each

online

retailer

Amazon.com

-LRB-

-RRB-

Education

High

School

marketshare

Amazon

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Region

West

Household

Size

marketshare

Amazon

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Region

South

Household

Size

marketshare

Amazon

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Household

Eldest

Age

ISP

Broadband

marketshare

Amazon

support

CI

-LRB-

-RRB-

Barnesandnoble.com

-LRB-

-RRB-

Education

Graduate

Household

Size

marketshare

BN

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Household

Eldest

Age

Income

marketshare

BN

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Region

South

Household

Size

Child

Yes

marketshare

BN

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Region

South

Household

Eldest

Age

marketshare

BN

support

CI

-LRB-

-RRB-

Research

Track

Paper

Ebay.com

-LRB-

-RRB-

Education

College

Region

South

marketshare

Ebay

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Education

College

Region

North

Central

marketshare

Ebay

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Region

South

Income

marketshare

Ebay

support

CI

-LRB-

-RRB-

-LRB-

-RRB-

Household

Eldest

Age

marketshare

Ebay

support

CI

-LRB-

-RRB-

Rule

-LRB-

-RRB-

for

Amazon.com

indicates

that

it

is

doing

particularly

well

in

households

with

middle-aged

heads

that

have

broadband

access

The

market

share

for

Amazon.com

in

this

segment

lies

significantly

outside

the

confidence

interval

computed

for

the

rule

On

the

other

hand

rule

-LRB-

-RRB-

for

Barnesandnoble.com

shows

that

they

are

doing

poorly

selling

to

a

segment

which

perhaps

represents

well

educated

couples

Given

that

this

is

a

large

segment

-LRB-

support

-RRB-

this

rule

suggests

that

they

could

try

and

examine

why

this

is

the

case

and

how

they

can

achieve

greater

penetration

in

this

segment

In

Ebay

's

case

all

four

rules

are

very

interesting

Rule

-LRB-

-RRB-

indicates

that

they

have

high

market

share

among

teenagers

while

rule

-LRB-

-RRB-

describes

a

segment

they

clearly

have

trouble

penetrating

For

many

other

categories

too

-LRB-

travel

and

home

electronics

in

particular

-RRB-

the

significant

SQ

rules

that

we

learned

were

highly

interesting

As

these

examples

suggest

these

rules

can

be

insightful

identify

interesting

segments

and

have

significant

business

potential

Varying

support

and

significance

levels

To

test

how

the

methods

perform

as

the

minimum

support

and

significance

levels

vary

for

one

site

we

generated

significant

SQ

rules

for

many

values

of

the

minimum

support

and

significance

level

parameters

Figures

and

show

how

the

number

of

significant

rules

and

the

false

discovery

rate

vary

with

support

As

the

minimum

support

threshold

is

lowered

the

number

of

significant

SQ

rules

discovered

increases

However

the

FDR

increases

as

the

support

threshold

is

lowered

suggesting

a

tradeoff

between

discovering

many

significant

rules

while

keeping

the

FDR

low

A

practical

outcome

is

that

it

may

be

desirable

to

have

higher

minimum

supports

-LRB-

to

keep

FDR

low

-RRB-

but

not

too

high

that

very

few

rules

are

discovered

Figures

and

illustrate

a

similar

tradeoff

for

the

significance

level

parameter

As

decreases

FDR

is

lower

but

this

results

in

fewer

number

of

significant

rules

being

discovered

Again

the

implication

is

that

it

may

be

desirable

to

have

a

low

-LRB-

to

keep

FDR

low

-RRB-

but

not

too

low

that

very

few

rules

are

discovered

support

of

s

i

gn

i

f

i

c

an

t

r

u

l

e

s

support

FD

R

Figure

Effect

of

support

on

of

rules

Figure

Effect

of

support

on

FDR

significance

level

FD

R

support

support

support

support

significance

level

o

f

s

i

g

n

ificant

r

u

le

s

support

support

support

support

Figure

Effect

of

on

FDR

Figure

Effect

of

on

of

rules

Research

Track

Paper

Summary

results

for

online

book

retailers

Based

on

this

general

tradeoff

we

chose

minimum

support

of

and

chose

an

of

in

order

to

report

summary

results

for

the

three

sites

Table

summarizes

the

number

of

significant

rules

discovered

and

the

false

discovery

rates

of

the

procedure

As

the

values

in

the

table

and

the

examples

above

show

our

procedure

can

be

used

effectively

to

learn

a

good

set

of

significant

SQ

rules

while

keeping

the

false

discovery

rates

reasonable

Table

Summary

of

results

Web

site

Significant

Rules

False

Discovery

Rate

Amazon

Barnesandnoble

Ebay

In

this

section

we

first

presented

compelling

examples

of

rules

discovered

that

illustrate

the

potential

of

learning

significant

market

share

rules

We

then

examined

how

the

number

of

significant

rules

discovered

and

the

false

discovery

rate

changes

with

the

support

and

significance

level

-LRB-

-RRB-

parameters

The

results

of

this

analysis

suggested

a

tradeoff

between

generating

significant

rules

and

keeping

the

false

discovery

rate

low

Based

on

this

tradeoff

we

identified

a

specific

value

of

the

support

and

significance

parameters

and

showed

the

number

of

rules

discovered

for

these

values

We

compare

our

work

with

the

literature

based

on

three

aspects

rule

structure

rule

significance

and

methodology

Rule

structure

Rule

discovery

methods

on

a

quantitative

dataset

can

be

traced

back

to

-LSB-

-RSB-

where

rules

of

the

form

x

A

x

y

B

y

are

discovered

-LSB-

-RSB-

extends

the

structure

to

be

conjunctions

of

multiple

conditions

on

both

antecedent

and

consequent

of

a

rule

and

proposes

their

discovery

method

based

on

the

Apriori

algorithm

-LSB-

-RSB-

Although

rules

in

-LSB-

-RSB-

are

important

partitions

like

y

B

y

for

continuous

attributes

on

the

RHS

of

a

rule

only

gives

partial

description

of

the

subset

satisfying

the

LHS

of

the

rule

and

partial

descriptions

sometimes

are

misleading

Observing

this

problem

-LSB-

-RSB-

introduces

a

new

structure

where

the

consequent

of

a

rule

is

Mean

-LRB-

D

X

-RRB-

or

Variance

-LRB-

D

X

-RRB-

to

summarize

the

behavior

of

the

subset

satisfying

the

antecedent

-LSB-

-RSB-

further

extends

the

form

of

the

consequent

of

the

rule

such

that

it

can

be

of

Min

-LRB-

D

X

-RRB-

Max

-LRB-

D

X

-RRB-

or

Sum

-LRB-

D

X

-RRB-

Our

rule

structure

is

based

on

prior

work

the

antecedent

is

conjunctions

of

conditions

while

the

consequent

can

be

any

aggregate

function

f

on

multiple

attributes

to

describe

the

behavior

of

the

subset

satisfying

the

antecedent

Rule

significance

Any

combination

of

attributes

with

conditions

can

potentially

form

a

rule

Researchers

use

different

measurements

e.g.

support

and

confidence

to

select

only

important

rules

from

all

possible

rules

Based

on

the

support

and

confidence

framework

many

metrics

have

been

developed

such

as

gain

-LSB-

-RSB-

conviction

-LSB-

-RSB-

unexpectedness

-LSB-

-RSB-

Although

these

metrics

can

be

generalized

to

rules

where

the

antecedent

and

consequent

are

both

conjunctions

of

the

form

value

Attribute

value

for

quantitative

datasets

they

are

not

applicable

for

rules

whose

consequent

is

a

function

such

as

Mean

-LRB-

D

X

-RRB-

or

in

general

f

-LRB-

D

X

-RRB-

To

solve

this

non-trivial

problem

we

use

statistical

significance

tests

to

evaluate

rules

so

that

the

consequent

of

a

rule

is

not

expected

by

chance

alone

In

the

data

mining

literature

statistical

significance

tests

are

commonly

used

in

many

applications

For

example

chi-square

-LRB-

-RRB-

is

a

statistic

to

test

correlations

between

attributes

in

binary

or

categorical

data

and

it

has

been

applied

to

discover

correlation

rules

-LSB-

-RSB-

actionable

rules

-LSB-

-RSB-

and

contrast

sets

-LSB-

-RSB-

For

sparse

data

-LSB-

-RSB-

employ

Fisher

's

Exact

Test

to

detect

anomaly

patterns

for

disease

outbreaks

As

mentioned

in

Section

these

two

tests

are

special

cases

of

our

significance

test

when

we

apply

our

significance

definition

to

categorical

data

For

quantitative

rules

in

-LSB-

-RSB-

the

authors

use

a

standard

Z-test

to

determine

the

significance

of

inequality

of

means

between

a

subset

D

X

and

its

complement

D

D

X

-LSB-

-RSB-

defines

a

new

measurement

impact

to

evaluate

quantitative

rules

where

impact

can

identify

those

groups

that

contribute

most

to

some

outcome

such

as

profits

or

costs

For

areas

other

than

rule

discovery

standard

Z-tests

with

log-linear

models

is

used

in

Exploratory

Data

Analysis

for

OLAP

data

cubes

-LSB-

-RSB-

Our

significance

test

is

different

from

the

above

primarily

because

-LRB-

i

-RRB-

our

significance

definition

is

applicable

to

any

user-defined

aggregate

function

f

-LRB-

D

X

-RRB-

and

-LRB-

ii

-RRB-

we

using

nonparametric

methods

to

construct

distributions

and

confidence

intervals

in

which

f

-LRB-

D

X

-RRB-

is

expected

from

random

effects

alone

Methodology

Nonparametric

statistics

is

philosophically

related

to

data

mining

in

that

both

methods

typically

make

no

assumptions

on

distributions

of

data

or

test

statistics

Even

with

known

distribution

of

a

statistic

nonparametric

methods

are

useful

to

estimate

parameters

of

the

distribution

-LSB-

-RSB-

Nonparametric

methods

are

widely

used

on

testing

models

that

are

built

from

data

as

earliest

in

-LSB-

-RSB-

the

author

uses

randomization

tests

to

tackle

a

model

overfitting

problem

-LSB-

-RSB-

compares

bootstrap

and

cross-validation

for

model

accuracy

estimation

for

decision

trees

-LSB-

-RSB-

use

permutation

tests

to

select

attributes

based

on

contingency

tables

Rule

discovery

is

to

learn

local

features

which

is

inherently

different

from

models

Although

we

have

seen

methods

using

parametric

hypothesis

testing

approach

to

learning

rules

from

dataset

-LSB-

-RSB-

no

prior

work

has

been

found

on

discovering

large

number

of

rules

based

on

nonparametric

significance

tests

The

problem

of

multiple

hypothesis

testing/multiple

comparison

is

well

known

in

rule

discovery

a

good

review

of

which

can

be

found

in

-LSB-

-RSB-

On

sparse

binary

data

-LSB-

-RSB-

shows

that

with

proper

support

and

confidence

control

very

few

false

rules

will

be

discovered

However

rule

discovery

on

quantitative

data

faces

much

more

complicated

challenges

and

conventional

p-value

adjustment

methods

can

not

be

directly

applied

To

solve

this

problem

we

employ

false

discovery

rate

-LSB-

-RSB-

metric

to

estimate

the

number

of

false

rules

discovered

due

to

testing

a

large

number

of

rules

In

data

mining

FDR

has

been

shown

useful

in

-LSB-

-RSB-

for

categorical

data

with

known

number

of

hypotheses

and

we

extend

it

to

quantitative

rules

with

resampling

methods

In

this

paper

we

defined

a

new

category

of

rules

SQ

rules

and

the

significance

of

SQ

rules

on

quantitative

data

Then

we

presented

a

permutation-based

algorithm

for

learning

significant

SQ

rules

Furthermore

we

show

how

an

explicit

false

discovery

rate

can

be

estimated

for

our

procedure

which

makes

the

approach

useful

from

a

practical

perspective

We

presented

experiments

in

which

we

discovered

market

share

rules

a

specific

Research

Track

Paper

type

of

SQ

rules

in

real

online

purchase

datasets

and

demonstrated

that

our

approach

can

be

used

to

learn

interesting

rules

from

data

We

would

also

like

to

point

out

that

it

is

possible

to

compute

the

false

discovery

rate

-LRB-

FDR

-RRB-

for

several

possible

significance

levels

in

an

efficient

manner

-LRB-

without

creating

permutation

datasets

for

each

significance

level

-RRB-

Although

a

detailed

presentation

of

this

is

beyond

the

scope

of

this

paper

in

the

appendix

we

provide

an

overview

of

how

this

can

be

done

One

main

advantage

of

being

able

to

do

this

is

that

significant

SQ

rules

can

be

discovered

at

a

chosen

significance

level

that

is

computed

from

some

desired

FDR

Hence

rather

than

just

estimating

FDR

we

may

be

able

to

discover

significant

rules

given

a

specific

FDR

However

this

needs

to

be

studied

in

greater

detail

in

future

work

-LSB-

-RSB-

Agrawal

R.

and

Srikant

R.

Fast

Algorithms

for

Mining

Association

Rules

in

Proceedings

of

the

International

Conference

on

Very

Large

Databases

Santiago

Chile

-LSB-

-RSB-

Aumann

Y.

and

Lindell

Y.

A

Statistical

Theory

for

Quantitative

Association

Rules

in

Proceedings

of

The

Fifth

ACM

SIGKDD

Int

l

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

San

Diego

CA

-LSB-

-RSB-

Bay

S.

D.

and

Pazzani

M.

J.

Detecting

Change

in

Categorical

Data

Mining

Contrast

Sets

in

Proceedings

of

the

Fifth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

San

Diego

CA

-LSB-

-RSB-

Benjamini

Y.

and

Hochberg

Y.

Controlling

the

False

Discovery

Rate

A

Practical

and

Powerful

Approach

to

Multiple

Testing

Journal

of

Royal

Statistical

Society

B

vol

iss

pp.

-LSB-

-RSB-

Bolton

R.

and

Adams

N.

An

Iterative

Hypothesis-Testing

Strategy

for

Pattern

Discovery

in

Proceedings

of

the

Ninth

ACM

SIGKDD

Int

l

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

Washington

DC

-LSB-

-RSB-

Bolton

R.

J.

and

Hand

D.

J.

Significance

Tests

for

Patterns

in

Continuous

Data

in

Proceedings

of

the

IEEE

International

Conference

on

Data

Mining

pp.

San

Jose

CA

-LSB-

-RSB-

Bolton

R.

J.

Hand

D.

J.

and

Adams

N.

M.

Determining

Hit

Rate

in

Pattern

Search

in

Pattern

Detection

and

Discovery

ESF

Exploratory

Workshop

pp.

London

UK

-LSB-

-RSB-

Brijs

T.

Swinnen

G.

Vanhoof

K.

and

Wets

G.

Using

Association

Rules

for

Product

Assortment

Decisions

Case

Study

in

Proceedings

of

the

Fifth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

San

Diego

CA

-LSB-

-RSB-

Brin

S.

Motwani

R.

and

Silverstein

C.

Beyond

Market

Baskets

Generalizing

Association

Rules

to

Correlations

in

Proceedings

of

the

ACM

SIGMOD/PODS

Joint

Conference

pp.

Tucson

AZ

-LSB-

-RSB-

Brin

S.

Motwani

R.

Ullman

J.

D.

and

Tsur

S.

Dynamic

Itemset

Counting

and

Implication

Rules

for

Market

Basket

Data

in

Proceedings

ACM

SIGMOD

International

Conference

on

Management

of

Data

-LRB-

SIGMOD

-RRB-

pp.

Tucson

AZ

-LSB-

-RSB-

Clark

P.

and

Niblett

T.

The

Induction

Algorithm

Machine

Learning

vol

pp.

-LSB-

-RSB-

Clearwater

S.

and

Provost

F.

A

Tool

for

Knowledge-Based

Induction

in

Procs

of

the

Second

International

IEEE

Conference

on

Tools

for

Artificial

Intelligence

pp.

-LSB-

-RSB-

Efron

B.

and

Tibshirani

R.

J.

An

Introduction

to

the

Bootstrap

New

York

NY

Chapman

Hall

-LSB-

-RSB-

Frank

E.

and

Witten

I.

H.

Using

a

Permutation

Test

for

Attribute

Selection

in

Decision

Trees

in

Proceedings

of

Int

l

Conference

on

Machine

Learning

pp.

-LSB-

-RSB-

Fukuda

T.

Morimoto

Y.

Morishita

S.

and

Tokuyama

T.

Data

Mining

Using

Two-Dimensional

Optimized

Association

Rules

Scheme

Algorithms

and

Visualization

in

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

-LRB-

SIGMOD

-RRB-

pp.

Montreal

Quebec

Canada

-LSB-

-RSB-

Good

P.

Permutation

Tests

A

Practical

Guide

to

Resampling

Methods

for

Testing

Hypotheses

Edition

New

York

Springer

-LSB-

-RSB-

Hsu

J.

C.

Multiple

Comparisons

Theory

and

Methods

London

UK

Chapman

Hall

-LSB-

-RSB-

Jensen

D.

Knowledge

Discovery

through

Induction

with

Randomization

Testing

in

Proceedings

of

the

Knowledge

Discovery

in

Databases

Workshop

pp.

Menlo

Park

-LSB-

-RSB-

Jensen

D.

and

Cohen

P.

R.

Multiple

Comparisons

in

Induction

Algorithms

Machine

Learning

vol

pp.

-LSB-

-RSB-

Kohavi

R.

A

Study

of

Cross-Validation

and

Bootstrap

for

Accuracy

Estimation

and

Model

Selection

in

Proceedings

of

the

Fourteenth

International

Joint

Conference

on

Artificial

Intelligence

pp.

San

Mateo

CA

-LSB-

-RSB-

Lee

Y.

Buchanan

B.

G.

and

Aronis

J.

M.

Knowledge-Based

Learning

in

Exploratory

Science

Learning

Rules

to

Predict

Rodent

Carcinogenicity

Machine

Learning

vol

pp.

-LSB-

-RSB-

Ling

C.

X.

and

Li

C.

Data

Mining

for

Direct

Marketing

Problems

and

Solutions

in

Proceedings

of

the

Fourth

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

New

York

NY

-LSB-

-RSB-

Liu

B.

Hsu

W.

and

Ma

Y.

Identifying

Non-Actionable

Association

Rules

in

Proceedings

of

the

Seventh

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

San

Francisco

CA

-LSB-

-RSB-

Mani

D.

R.

Drew

J.

Betz

A.

and

Datta

P.

Statistics

and

Data

Mining

Techniques

for

Lifetime

Value

Modeling

in

Proceedings

of

the

Fifth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp

San

Diego

CA

-LSB-

-RSB-

Megiddo

N.

and

Srikant

R.

Discovering

Predictive

Association

Rules

in

Proceedings

of

the

Fourth

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

New

York

NY

-LSB-

-RSB-

Oates

T.

and

Jensen

D.

Large

Datasets

Lead

to

Overly

Complex

Models

An

Explanation

and

a

Solution

in

Proceedings

of

the

Fourth

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

Menlo

Park

CA

-LSB-

-RSB-

Padmanabhan

B.

and

Tuzhilin

A.

A

Belief-Driven

Method

for

Discovering

Unexpected

Patterns

in

Proceedings

of

the

Fourth

International

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

New

York

NY

Research

Track

Paper

-LSB-

-RSB-

Padmanabhan

B.

and

Tuzhilin

A.

Small

Is

Beautiful

Discovering

the

Minimal

Set

of

Unexpected

Patterns

in

Proceedings

of

the

Sixth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

Data

Mining

pp.

Boston

MA

-LSB-

-RSB-

Piatesky-Shapiro

G.

Discovery

Analysis

and

Presentation

of

Strong

Rules

in

Knowledge

Discovery

in

Databases

Piatesky-Shapiro

G.

and

Frawley

W.

J.

Eds

Menlo

Park

CA

AAAI/MIT

Press

pp.

-LSB-

-RSB-

Sarawagi

S.

Agrawal

R.

and

Megiddo

N.

Discovery-Driven

Exploration

of

Olap

Data

Cubes

in

Proceedings

of

the

Sixth

International

Conference

on

Extending

Database

Technology

-LRB-

EDBT

-RRB-

pp.

Valencia

Spain

-LSB-

-RSB-

Srikant

R.

and

Agrawal

R.

Mining

Quantitative

Association

Rules

in

Large

Relational

Tables

in

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

-LSB-

-RSB-

Webb

G.

Butler

S.

and

Newlands

D.

On

Detecting

Differences

between

Groups

in

Proceedings

of

the

Ninth

ACM

SIGKDD

Int

l

Conference

on

Knowledge

Discovery

and

Data

Mining

pp.

Washington

DC

-LSB-

-RSB-

Webb

G.

I.

Discovering

Associations

with

Numeric

Variables

in

Proceedings

of

The

Seventh

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

San

Francisco

CA

-LSB-

-RSB-

Westfall

P.

H.

and

Young

S.

S.

Resampling-Based

Multiple

Testing

Examples

and

Methods

for

P-Value

Adjustment

New

York

NY

John

Wiley

Sons

Inc

-LSB-

-RSB-

Wong

W.-K.

Moore

A.

Cooper

G.

and

Wagner

M.

Rule-Based

Anomaly

Pattern

Detection

for

Detecting

Disease

Outbreaks

in

Proceedings

of

the

Eighteenth

National

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

Edmonton

Canada

-LSB-

-RSB-

Wong

W.-K.

Moore

A.

Cooper

G.

and

Wagner

M.

Bayesian

Network

Anomaly

Pattern

Detection

for

Disease

Outbreaks

in

Proceedings

of

the

Twentieth

International

Conference

on

Machine

Learning

-LRB-

-RRB-

Washington

DC

APPENDIX

Discovering

false

discovery

rates

for

multiple

significance

levels

Let

us

continue

to

use

the

example

N

perm

and

On

the

dataset

D

from

the

algorithm

SigSQrules

we

generate

significant

rules

as

well

as

each

rule

's

p-value

Because

there

are

N

perm

values

in

each

distribution

the

smallest

possible

p-value

from

the

permutation

tests

is

-LRB-

N

perm

-RRB-

and

all

possible

p-values

are

S

-LCB-

-LRB-

N

perm

-RRB-

-LRB-

N

perm

-RRB-

-RCB-

Let

N

sig

-LSB-

ind

-RSB-

be

the

number

of

significant

rules

whose

pvalue

is

no

larger

than

ind

S.

For

example

if

there

are

rules

whose

p-value

and

rules

whose

p-value

then

N

sig

-LSB-

-RSB-

and

N

sig

-LSB-

-RSB-

Without

further

permutation

tests

with

N

sig

-LSB-

-RSB-

we

know

how

many

rules

will

be

discovered

if

we

lower

the

significance

level

from

to

ind

For

example

if

ind

there

are

only

N

sig

-LSB-

-RSB-

rules

whose

p-value

is

no

larger

than

ind

therefore

we

expect

to

discover

rules

Similarly

for

each

permutation

dataset

D

at

each

significance

level

ind

we

can

compute

the

number

of

significant

rules

and

their

p-values

by

applying

SigSQrules

only

once

Note

that

all

discoveries

from

D

are

false

discoveries

because

the

relationships

between

A

and

B

are

removed

Let

N

sig-perm

-LSB-

i

-RSB-

-LSB-

ind

-RSB-

be

the

number

of

discoveries

from

permutation

datasets

D

-LSB-

i

-RSB-

For

example

N

sig-perm

-LSB-

-RSB-

-LSB-

-RSB-

means

we

have

discoveries

from

the

permutation

dataset

D

-LSB-

-RSB-

at

ind

We

implement

this

procedure

on

multiple

permutation

datasets

and

Median

-LRB-

N

sig-perm

-LSB-

-RSB-

-LSB-

ind

-RSB-

-RRB-

is

the

estimate

of

false

discoveries

at

each

significance

level

ind

on

permutation

datasets

Therefore

FDR

-LRB-

ind

-RRB-

Median

-LRB-

N

sig-perm

-LSB-

-RSB-

-LSB-

ind

-RSB-

-RRB-

N

sig

-LSB-

ind

-RSB-

We

use

Median

to

estimate

the

expectation

which

conforms

to

nonparametric

statistical

considerations

-LRB-

median

is

the

best

estimator

for

expectation

when

the

underlying

distribution

is

unknown

-RRB-

Empirically

we

showed

in

Figure

that

FDR

-LRB-

ind

-RRB-

is

an

increasing

function

on

ind

It

means

that

by

decreasing

ind

we

can

control

FDR

-LRB-

ind

-RRB-

to

a

smaller

value

We

are

not

always

guaranteed

though

to

be

able

to

set

an

individual

significance

level

such

that

FDR

It

is

possible

that

even

when

we

decrease

ind

to

a

level

that

almost

no

rules

are

discovered

FDR

is

still

much

larger

than

In

other

words

there

are

always

a

large

proportion

of

spurious

rules

discovered

from

some

datasets

For

example

if

attributes

independent

based

on

a

test

statistic

then

Median

-LRB-

N

sig-perm

-LSB-

-RSB-

-LSB-

ind

-RSB-

-RRB-

N

sig

-LSB-

ind

-RSB-

for

all

significance

levels

and

FDR

We

want

to

point

out

that

this

is

a

desirable

property

of

our

method

on

controlling

FDR

because

there

are

many

real-world

datasets

whose

attributes

are

truly

independent

from

each

other

Traditional

methods

can

not

estimate

how

many

rules

should

be

discovered

but

with

our

technique

we

can

draw

the

conclusion

that

there

is

no

rule

to

be

discovered

because

of

the

rules

is

better

than

chance

This

nonparametric

method

to

estimate

and

control

FDR

is

applicable

to

quantitative

datasets

and

broad

types

of

rules

Research

Track

Paper

market share rules
nonparametric methods
resampling
rule discovery
statistical quantitative rules