Integration

of

Information

Assurance

and

Security

into

the

Model

Curriculum

In

this

paper

we

present

the

context

of

the

work

of

the

Curriculum

Committee

on

the

IT

curriculum

volume

described

in

the

Overview

Draft

document

of

the

Joint

Task

Force

for

Computing

Curriculum

We

also

provide

a

brief

introduction

to

the

history

and

work

of

the

Information

Assurance

Education

community

These

two

perspectives

provide

the

foundation

for

the

main

thrust

of

the

paper

which

is

a

description

of

the

Information

Assurance

and

Security

-LRB-

IAS

-RRB-

component

of

the

document

Finally

we

end

the

paper

with

an

example

of

how

IAS

is

being

implemented

at

BYU

as

a

pervasive

theme

that

is

woven

throughout

the

curriculum

and

conclude

with

some

observations

about

the

first

year

's

experience

K.

-LSB-

Computer

and

Information

Science

Education

-RSB-

Accreditation

General

Terms

Management

Security

In

December

a

meeting

-LRB-

-RRB-

of

interested

parties

from

fifteen

four-year

IT

programs

from

the

US

along

with

representatives

from

IEEE

ACM

and

ABET

began

work

on

the

formalization

of

Information

Technology

as

an

accredited

academic

discipline

The

effort

has

evolved

into

SIGITE

the

ACM

SIG

for

Information

Technology

Education

During

this

evolution

three

main

efforts

have

proceeded

in

parallel

-RRB-

Definition

of

accreditation

standards

for

IT

programs

-RRB-

Creation

of

a

model

curriculum

for

four-year

IT

programs

and

-RRB-

Description

of

the

characteristics

that

distinguish

IT

programs

from

the

sister

disciplines

in

computing

One

of

the

biggest

challenges

during

the

creation

of

the

model

curriculum

was

understanding

and

presenting

the

knowledge

area

that

was

originally

called

security

Some

of

us

were

uncomfortable

with

the

term

because

it

was

not

broad

enough

to

cover

the

range

of

concepts

that

we

felt

needed

to

be

covered

We

became

aware

of

a

community

that

had

resolved

many

of

the

issues

associated

with

the

broader

context

we

were

seeking

Information

Assurance

Information

assurance

has

been

defined

as

a

set

of

measures

intended

to

protect

and

defend

information

and

information

systems

by

ensuring

their

availability

integrity

authentication

confidentiality

and

non-repudiation

This

includes

providing

for

restoration

of

information

systems

by

incorporating

protection

detection

and

reaction

capabilities

The

IA

community

and

work

done

by

IA

educators

became

useful

in

defining

requisite

security

knowledge

for

information

technology

education

programs

We

believe

that

the

Information

Technology

and

the

Information

Assurance

Education

communities

have

much

to

share

At

the

th

Colloquium

for

Information

System

Security

Education

in

Atlanta

we

introduced

and

to

the

IA

Education

community

-LSB-

-RSB-

In

the

current

paper

we

introduce

the

history

and

current

state

of

IA

education

to

the

SIGITE

community

In

addition

we

demonstrate

how

significant

concepts

from

the

Information

Assurance

community

have

been

integrated

into

and

In

the

first

week

of

December

of

representatives

from

undergraduate

information

technology

-LRB-

IT

-RRB-

programs

from

across

the

country

gathered

together

near

Provo

Utah

to

develop

a

community

and

begin

to

establish

academic

standards

for

this

rapidly

growing

discipline

This

first

Conference

on

Information

Technology

Curriculum

-LRB-

-RRB-

was

also

attended

by

representatives

from

two

professional

societies

the

Association

for

Computing

Machinery

-LRB-

ACM

-RRB-

and

the

Institute

of

Electrical

and

Electronics

Engineers

Inc.

-LRB-

IEEE

-RRB-

and

also

the

Accreditation

Board

for

Engineering

and

Technology

Inc.

-LRB-

ABET

-RRB-

This

invitational

conference

was

the

culmination

of

an

effort

begun

several

months

earlier

by

five

of

these

universities

who

had

formed

a

steering

committee

to

organize

a

response

from

existing

IT

programs

to

several

initiatives

to

define

the

academic

discipline

of

IT

The

steering

committee

wanted

to

ensure

that

the

input

of

existing

programs

played

a

significant

role

in

the

definition

of

the

field

A

formal

society

and

three

main

committees

were

formed

by

the

attendees

of

The

society

was

the

Society

for

Information

Technology

Education

-LRB-

SITE

-RRB-

one

of

the

committees

formed

was

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

SIGITE

October

Newark

New

Jersey

USA

Copyright

ACM

the

executive

board

for

SITE

composed

of

a

president

vice-president

secretary

treasurer

regional

representatives

and

an

activities

chairperson

The

other

two

committees

formed

were

the

IT

Curriculum

Committee

including

subcommittees

for

and

programs

and

the

IT

Accreditation

Committee

also

including

subcommittees

for

and

programs

The

development

of

IT

as

an

academic

discipline

is

similar

to

the

process

that

computer

science

-LRB-

CS

-RRB-

went

through

in

the

's

and

's

In

fact

looking

at

the

placement

of

CS

programs

in

academic

institutions

around

the

U.S.

illustrates

the

debate

that

swirled

around

the

discipline

as

its

core

was

being

defined

Some

CS

programs

are

in

departments

of

mathematics

others

are

in

engineering

schools

and

many

others

have

become

mainstay

programs

within

newly

emerging

colleges

of

computing

Information

technology

as

it

is

practiced

at

this

moment

in

its

evolution

reflects

similar

growing

pains

IT

programs

exist

in

colleges

of

computing

in

CS

departments

in

schools

of

technology

and

in

business

schools

Professors

of

information

technology

possess

degrees

in

information

systems

electronics

communications

graphics

arts

economics

mathematics

computer

science

and

other

disciplines

Few

to

of

them

have

a

degree

in

information

technology

It

should

be

acknowledged

here

that

IT

has

two

substantially

different

interpretations

and

that

these

should

be

clarified

Information

Technology

-LRB-

IT

-RRB-

in

its

broadest

sense

encompasses

all

aspects

of

computing

technology

IT

as

an

academic

discipline

focuses

on

meeting

the

needs

of

users

within

an

organizational

and

societal

context

through

the

selection

creation

application

integration

and

administration

of

computing

technologies

A

more

detailed

history

of

SIGITE

is

available

in

-LSB-

-RSB-

SIGITE

is

directly

involved

with

the

Joint

Task

Force

for

Computing

Curriculum

and

has

representatives

on

the

task

force

This

task

force

is

a

continuation

of

the

effort

that

created

-LSB-

-RSB-

the

current

computer

science

curriculum

standard

has

been

relabeled

and

the

current

draft

of

the

Overview

document

-LSB-

-RSB-

presents

the

structure

being

used

to

describe

computing

and

its

sub-disciplines

-LRB-

See

Figure

-RRB-

The

SIGITE

Curriculum

Committee

is

responsible

for

the

Information

Technology

Curriculum

Volume

was

made

available

for

comment

in

mid

Figure

Information

Assurance

Education

Information

assurance

has

been

defined

as

a

set

of

measures

intended

to

protect

and

defend

information

and

information

systems

by

ensuring

their

availability

integrity

authentication

confidentiality

and

non-repudiation

This

includes

providing

for

restoration

of

information

systems

by

incorporating

protection

detection

and

reaction

capabilities

-LRB-

National

Security

Agency

-RRB-

-LSB-

-RSB-

Information

assurance

education

then

includes

all

efforts

to

prepare

a

workforce

with

the

needed

knowledge

skills

and

abilities

to

assure

our

information

systems

especially

critical

national

security

systems

Information

assurance

education

has

been

growing

in

importance

and

activity

for

the

past

two

decades

A

brief

look

at

the

involved

entities

and

history

will

shed

light

on

the

growth

The

National

Information

Assurance

Education

and

Training

Partnership

-LRB-

NIETP

-RRB-

program

is

a

partnership

among

government

academia

and

industry

focused

on

advancing

information

assurance

education

training

and

awareness

The

NIETP

was

initiated

in

under

National

Security

Directive

and

has

since

been

reauthorized

several

times

The

NIETP

serves

in

the

capacity

of

national

manager

for

information

assurance

education

and

training

related

to

national

security

systems

and

coordinates

this

effort

with

the

Committee

on

National

Security

Systems

-LRB-

CNSS

-RRB-

The

CNSS

provides

a

forum

for

the

discussion

of

policy

issues

sets

national

policy

and

promulgates

direction

operational

procedures

and

guidance

for

the

security

of

national

security

systems

National

security

systems

are

information

systems

operated

by

the

U.S.

Government

its

contractors

or

agents

that

contain

classified

information

or

that

involve

intelligence

activities

involve

cryptographic

activities

related

to

national

security

involve

command

and

control

of

military

forces

involve

equipment

that

is

an

integral

part

of

a

weapon

or

weapons

system

-LRB-

s

-RRB-

or

are

critical

to

the

direct

fulfillment

of

military

or

intelligence

missions

-LRB-

not

including

routine

administrative

and

business

applications

-RRB-

-LSB-

-RSB-

CNSS

is

responsible

for

the

development

of

principles

policies

guidelines

and

standards

that

concern

systems

holding

or

related

to

national

security

information

Education

and

training

standards

are

among

the

many

standards

and

guidelines

that

CNSS

issues

The

training/education

standards

issued

to

date

include

a

-RRB-

NSTISSI

The

National

Training

Standard

for

Information

Systems

Security

Professionals

b

-RRB-

CNSSI

The

National

Information

Assurance

Training

Standard

for

Senior

Systems

Managers

c

-RRB-

CNSSI

The

National

Information

Assurance

Training

Standard

for

System

Administrators

d

-RRB-

CNSSI

Information

Assurance

Training

Standard

for

Information

Systems

Security

Officers

and

e

-RRB-

NSTISSI

The

National

Training

Standard

for

Systems

Certifiers

CNSSI

The

National

Under

Executive

Order

-LRB-

E.O.

-RRB-

of

October

Critical

Infrastructure

Protection

in

the

Information

Age

the

President

redesigned

the

National

Security

Telecommunications

and

Information

Systems

Security

Committee

-LRB-

NSTISSC

-RRB-

as

the

Committee

on

National

Security

Systems

-LRB-

CNSS

-RRB-

Training

Standard

for

Information

Security

Risk

Analysts

will

be

released

soon

The

NSTISSI-CNSSI

standards

referenced

above

have

been

used

to

develop

in-service

training

and

education

opportunities

for

enlisted

and

civilian

employees

in

an

effort

to

assure

quality

preparation

of

professionals

entrusted

with

securing

our

critical

information

In

addition

to

providing

a

basis

for

in-service

education

and

training

the

NSTISSI-CNSSI

standards

have

also

been

deployed

to

colleges

and

universities

in

an

effort

to

also

prepare

qualified

individuals

preservice

The

most

significant

effort

to

involve

colleges

and

universities

has

been

through

the

National

Centers

of

Academic

Excellence

in

Information

Assurance

Education

-LRB-

CAEIAE

-RRB-

Program

The

CAEIAE

program

was

started

in

by

the

National

Security

Agency

-LRB-

NSA

-RRB-

and

is

now

jointly

sponsored

by

the

NSA

and

the

Department

of

Homeland

Security

-LRB-

DHS

-RRB-

in

support

of

the

President

's

National

Strategy

to

Secure

Cyberspace

February

The

purpose

of

the

program

is

to

recognize

colleges

and

universities

for

their

efforts

in

information

assurance

education

and

also

to

encourage

more

colleges

and

universities

to

develop

courses

and

programs

of

study

in

information

assurance

In

order

to

be

eligible

to

apply

for

CAEIAE

certification

an

institution

must

first

demonstrate

that

it

teaches

the

content

covered

in

NSTISSI

The

National

Training

Standard

for

Information

Systems

Security

Professionals

Once

an

institution

has

been

certified

it

is

eligible

to

apply

for

CAEIAE

status

Criteria

for

becoming

a

CAEIAE

include

the

following

a

-RRB-

evidence

of

partnerships

in

IA

education

b

-RRB-

IA

must

be

treated

as

a

multidisciplinary

science

c

-RRB-

evidence

that

the

university

encourages

the

practice

of

information

assurance

in

its

operations

d

-RRB-

demonstration

of

information

assurance

research

e

-RRB-

demonstration

that

the

IA

curriculum

reaches

beyond

physical

geographic

borders

f

-RRB-

evidence

of

faculty

productivity

in

information

assurance

research

and

scholarship

g

-RRB-

demonstration

of

state

of

the

art

information

assurance

resources

h

-RRB-

a

declared

concentration

-LRB-

s

-RRB-

in

information

assurance

i

-RRB-

a

university

recognized

center

in

information

assurance

and

j

-RRB-

dedicated

information

assurance

faculty

-LRB-

-RRB-

-LSB-

-RSB-

In

there

were

seven

institutions

designated

as

the

inaugural

CAEIAE

schools

The

certification

is

good

for

three

years

at

which

time

institutions

can

reapply

Annually

an

additional

institutions

are

awarded

the

certification

today

there

are

more

than

CAEIAE

institutions

The

types

of

institutions

and

programs

that

are

applying

and

being

certified

are

growing

not

just

in

number

but

also

in

diversity

In

the

first

round

of

certification

the

institutions

were

largely

research

institutions

and

their

respective

programs

were

at

the

graduate

level

in

computer

science

Today

institutions

are

certifying

courses

at

the

undergraduate

level

in

computer

science

management

information

systems

and

information

technology

The

work

being

done

by

SIGITE

is

important

to

the

further

expansion

of

information

assurance

education

as

information

assurance

expands

beyond

the

development

of

information

systems

to

include

the

entire

system

life

cycle

including

deployment

operation

maintenance

a

retirement

of

such

systems

The

volume

is

modeled

on

It

consists

of

chapters

and

appendices

The

current

draft

resides

at

-LSB-

-RSB-

Chapter

Introduction

Chapter

Lessons

from

Past

Reports

Chapter

Changes

in

the

Information

Technology

Discipline

Chapter

Principles

Chapter

Overview

of

the

IT

Body

of

Knowledge

Chapter

Overview

of

the

Curricular

Models

Chapter

The

Core

of

the

Curriculum

Chapter

Completing

the

Curriculum

Chapter

Professional

Practice

Chapter

Characteristics

of

IT

Graduates

Chapter

Computing

across

the

Curriculum

Chapter

Institutional

Challenges

Acknowledgements

Bibliography

Appendix

A

The

IT

Body

of

Knowledge

Appendix

B

IT

Course

Descriptions

Chapters

and

are

of

particular

interest

for

this

discussion

Chapter

is

an

overview

of

the

IT

body

of

knowledge

A

summary

is

included

as

Appendix

A.

Chapter

discusses

the

relationship

of

the

core

topics

described

in

the

body

of

knowledge

to

IT

curriculum

IAS

is

explicitly

mentioned

in

three

contexts

Section

as

part

of

the

IT

Fundamentals

Knowledge

Area

-LRB-

KA

-RRB-

Section

as

a

pervasive

theme

Section

as

a

KA

that

integrates

the

IAS

concepts

for

students

ready

to

graduate

IAS

is

the

only

area

that

is

an

IT

Fundamental

a

pervasive

theme

and

also

a

complete

KA

with

a

recommended

senior

level

course

for

integrating

all

of

the

concepts

Clearly

presents

Information

Assurance

and

Security

as

a

core

competency

required

by

every

graduate

of

an

IT

program

During

the

early

analysis

of

IT

as

an

academic

discipline

Delphi

studies

were

performed

that

ranked

Security

as

a

central

area

for

IT

-LSB-

-RSB-

As

we

studied

the

issues

several

members

of

the

committees

involved

were

uncomfortable

with

security

as

the

name

for

the

knowledge

area

The

name

seemed

too

restrictive

At

the

annual

SIGITE

conference

in

two

of

the

authors

were

introduced

to

the

other

author

and

the

Center

for

Research

and

Education

in

Information

Assurance

and

Security

-LRB-

Cerias

-RRB-

at

Purdue

The

BYU

faculty

was

dissatisfied

with

the

security

component

in

the

IT

curriculum

and

the

SIGITE

curriculum

committee

was

struggling

with

the

Security

KA

for

Through

flyers

at

the

conference

we

became

aware

of

the

Information

Assurance

Education

Graduate

Certificate

-LRB-

IAEGC

-RRB-

program

funded

by

the

NSA

With

encouragement

from

colleagues

and

the

administration

of

the

School

of

Technology

the

primary

author

attended

the

program

The

experience

has

had

a

significant

impact

on

and

the

BYU

curriculum

We

discovered

that

NSA

had

begun

to

use

the

umbrella

term

Information

Assurance

-LSB-

-RSB-

to

cover

what

we

were

calling

security

Even

though

this

term

is

defined

to

cover

exactly

what

the

IT

community

meant

by

security

the

use

of

the

terminology

elicited

a

lot

of

blank

stares

We

found

that

explicitly

adding

security

to

the

name

of

the

knowledge

area

eliminated

much

of

the

confusion

We

are

indebted

to

the

Center

for

Education

and

Research

for

Information

Assurance

and

Security

-LRB-

CERIAS

-RRB-

-LSB-

-RSB-

at

Purdue

whose

name

provided

the

inspiration

to

use

IAS

as

a

name

for

the

knowledge

area

Once

the

naming

issue

was

resolved

the

SIGITE

curriculum

committee

struggled

to

find

a

model

for

IAS

that

could

be

understood

by

freshman

IT

students

provide

a

framework

to

integrate

IAS

concepts

that

are

integrated

into

nearly

all

of

the

other

KAs

be

rich

enough

to

support

a

senior

level

course

that

ties

everything

together

When

A

Model

for

Information

Assurance

An

Integrative

Approach

-LSB-

-RSB-

was

discovered

the

writing

committee

achieved

consensus

on

a

model

The

cube

-LRB-

see

Figure

-RRB-

provides

a

simple

visual

representation

that

a

freshman

can

understand

yet

the

dimensional

structure

facilitates

the

detailed

analysis

required

for

use

in

technology

specific

contexts

and

is

comprehensive

enough

to

encompass

a

capstone

learning

experience

Figure

uses

this

model

to

structure

IAS

concepts

throughout

the

document

SECTION

header

RECOMMENDATIONS

FOR

PERVASIVE

THEMES

IN

During

the

deliberations

of

the

SIGITE

Curriculum

Committee

several

topics

emerged

that

were

considered

essential

but

that

did

not

seem

to

belong

in

a

single

specific

knowledge

area

or

unit

These

topics

referred

to

as

pervasive

themes

are

user

advocacy

information

assurance

and

security

ethics

and

professional

responsibility

the

ability

to

manage

complexity

through

abstraction

modeling

best

practices

patterns

standards

and

the

use

of

appropriate

tools

a

deep

understanding

of

information

and

communication

technologies

and

their

associated

tools

adaptability

life-long

learning

and

professional

development

interpersonal

skills

The

committee

states

that

these

topics

are

best

addressed

multiple

times

in

multiple

classes

beginning

in

the

IT

fundamentals

class

and

woven

like

threads

throughout

the

tapestry

of

the

IT

curriculum

-LSB-

-RSB-

These

themes

need

to

be

made

explicit

in

the

minds

of

the

students

and

the

faculty

The

themes

touch

many

of

the

topics

throughout

the

curriculum

Every

time

a

new

technology

is

announced

in

the

media

an

instructor

has

the

opportunity

to

drive

home

the

importance

of

life-long

learning

Every

time

there

is

a

cyber-crime

in

the

media

we

have

the

opportunity

to

discuss

the

ethical

and

professional

ramifications

It

is

recommended

that

an

IT

Fundamentals

course

be

taught

early

in

the

curriculum

where

all

of

these

themes

are

introduced

and

discussed

as

concepts

that

touch

everything

an

IT

professional

does

Each

of

these

topics

deserves

a

full

treatment

however

for

the

purposes

of

this

paper

we

will

focus

on

IAS

possibly

the

most

pervasive

theme

We

will

address

one

approach

to

achieve

addressing

IAS

multiple

times

in

multiple

classes

in

section

below

In

early

the

SIGITE

curriculum

committee

divided

into

working

groups

around

the

knowledge

areas

defined

by

-LSB-

-RSB-

to

make

an

initial

cut

at

the

list

of

topics

for

each

KA

A

significant

revision

was

accomplished

and

reviewed

by

the

participants

at

the

IAEGC

program

at

Purdue

in

August

The

list

of

areas

for

the

IAS

KA

was

finalized

in

late

at

a

full

IT

Curriculum

Committee

meeting

The

draft

of

the

completed

IAS

KA

was

completed

in

early

Feb

by

the

IAS

working

group

edited

by

the

writing

committee

in

late

Feb

and

was

presented

to

the

full

committee

in

April

Figure

is

a

list

of

the

IAS

KA

and

its

areas

The

basic

structure

and

vocabulary

is

derived

directly

from

work

done

in

the

IA

community

specifically

Maconachy

et

al.

-LSB-

-RSB-

The

number

in

parenthesis

is

the

number

of

lecture

hours

the

committee

thought

would

be

required

to

give

an

IT

student

minimum

exposure

to

the

unit

It

should

be

noted

that

the

ordering

of

units

in

all

of

the

KAs

is

first

Fundamentals

if

there

is

one

and

then

the

units

are

sorted

in

order

of

the

number

of

core

hours

This

ordering

should

not

be

considered

as

any

indication

of

the

order

the

units

would

be

covered

pedagogically

in

an

implemented

curriculum

Figure

A

summary

of

the

IAS

KA

is

in

Appendix

A

and

a

complete

treatment

is

found

in

-LSB-

-RSB-

including

topics

core

learning

outcomes

and

example

elective

learning

outcomes

In

reviewing

this

model

curriculum

for

IAS

in

Information

Technology

it

should

be

remembered

that

the

core

topics

and

associated

lecture

hours

are

the

minimum

coverage

that

every

IT

student

in

every

program

should

receive

We

would

expect

that

most

institutions

would

provide

additional

instruction

in

Information

Assurance

and

Security

according

to

the

strengths/areas

of

specialization

in

their

programs

of

study

The

Information

Technology

program

at

BYU

began

officially

in

Fall

with

a

faculty

consisting

of

Two

electronics

engineering

technology

-LRB-

EET

-RRB-

professors

who

were

instrumental

in

the

evolution

of

the

existing

EET

program

at

BYU

into

an

IT

program

One

electrical

engineering

Ph.D.

newly

arrived

from

the

aerospace

industry

One

computer

science

instructor

who

had

done

part

time

teaching

and

had

been

part

of

the

department

for

year

with

several

years

in

system

development

in

health

care

One

computer

science

Ph.D.

with

recent

executive

management

responsibilities

in

network

hardware

and

service

provider

businesses

The

former

department

chair

of

the

technology

education

program

for

secondary

schools

joined

in

One

computer

science

Ph.

D.

with

extensive

industry

experience

in

data

privacy

and

IT

management

joined

in

This

is

obviously

a

diverse

group

of

people

each

of

whom

joined

the

department

because

they

thought

that

the

existing

computing

programs

at

BYU

did

not

offer

students

preparation

for

the

practical

aspects

of

system

delivery

to

customers

We

are

evenly

divided

between

long-term

academics

and

recent

retreads

from

industry

However

the

academics

have

also

each

had

significant

industrial

experience

which

provided

the

motivation

for

them

to

accept

positions

in

the

new

IT

program

The

BYU

curriculum

began

as

a

traditional

stovepipe

approach

of

courses

oriented

around

topics

like

networking

databases

and

operating

systems

borrowed

from

CS

EET

CE

and

IS

and

evolved

to

a

more

integrated

approach

starting

at

the

introductory

levels

so

that

advanced

topic

oriented

courses

are

more

easily

sequenced

We

have

also

discovered

that

the

integrative

nature

of

IT

forces

a

focus

on

the

seams

between

technologies

rather

than

implementation

of

components

This

fundamental

difference

in

focus

is

one

of

the

primary

differences

that

distinguishes

IT

from

other

computing

disciplines

that

focus

on

the

design

and

implementation

of

components

-LSB-

-RSB-

-LSB-

-RSB-

Over

the

last

years

BYU

faculty

has

participated

actively

in

SIGITE

and

attempted

to

share

what

has

been

learned

with

the

emerging

IT

community

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

The

BYU

curriculum

has

evolved

into

what

calls

a

core/integration

first

approach

-LSB-

-RSB-

Significant

portions

of

the

introductory

material

in

operating

systems

databases

web

systems

networking

had

been

moved

to

lower

division

courses

by

early

Much

of

the

shift

occurred

when

the

introduction

to

web

systems

was

moved

from

the

junior

to

the

sophomore

year

and

introductory

material

sufficient

to

understand

web

systems

was

included

for

networking

databases

operating

system

administration

and

OS

process

models

The

improvements

in

flow

and

reduced

redundancy

have

been

noticeable

in

the

upper

division

core

courses

Appendix

B

graphs

the

current

BYU

course

structure

In

late

and

early

we

began

implementing

the

pervasive

theme

of

IAS

in

earnest

A

senior

level

IAS

class

had

been

introduced

into

the

curriculum

in

early

and

was

made

a

requirement

in

However

we

recognized

that

simply

adding

a

required

course

at

the

end

of

a

student

's

college

experience

would

not

be

adequate

SIGITE

discussions

had

placed

security

in

the

pervasive

theme

category

at

the

very

beginning

though

the

name

of

the

KA

was

n't

chosen

until

We

were

faced

with

the

challenge

of

integrating

the

IAS

fundamentals

into

the

introductory

courses

morphing

the

security

modules

in

the

existing

classes

to

use

the

MSRW

-LSB-

-RSB-

framework

and

bringing

all

of

the

students

in

the

program

up

to

speed

on

the

new

framework

simultaneously

Our

approach

has

been

to

prepare

one

hour

modules

on

the

MSRW

framework

that

can

be

used

in

an

existing

course

to

bring

students

up

to

speed

or

taught

in

seminars

as

needed

We

are

in

the

process

of

integrating

the

IAS

Fundamentals

into

our

introductory

courses

We

successfully

integrated

the

IAS

modules

into

the

sophomore

introduction

to

web-based

systems

course

which

was

already

introducing

all

of

the

major

IT

areas

The

course

was

modified

to

replace

a

week

team

project

experience

with

a

week

team

oriented

lab

and

then

using

the

time

for

IAS

topics

Much

remains

to

be

done

but

the

initial

experience

is

positive

The

faculty

seems

unified

in

their

desire

to

implement

IAS

as

a

pervasive

theme

For

example

lecture

and

lab

hours

are

now

included

in

the

computer

communications

course

lecture

hours

and

lab

hours

were

added

to

the

web

systems

course

The

IAS

component

of

the

database

course

was

rearranged

and

strengthened

with

lecture

hour

added

Similar

adjustments

have

been

made

throughout

the

curriculum

IAS

Information

Assurance

and

Security

-LRB-

core

hours

-RRB-

Fundamental

Aspects

-LRB-

-RRB-

Security

Mechanisms

-LRB-

countermeasures

-RRB-

-LRB-

-RRB-

Operational

Issues

-LRB-

-RRB-

Policy

-LRB-

-RRB-

Attacks

-LRB-

-RRB-

Security

Domains

-LRB-

-RRB-

Forensics

-LRB-

-RRB-

Information

States

-LRB-

-RRB-

Security

Services

-LRB-

-RRB-

Threat

Analysis

Model

-LRB-

-RRB-

Vulnerabilities

-LRB-

-RRB-

In

addition

to

improving

the

IAS

component

of

the

BYU

curriculum

we

have

done

an

analysis

of

our

coverage

of

the

proposed

core

We

have

several

adjustments

in

other

parts

of

our

curriculum

Since

we

evolved

from

an

EET

program

the

hardware

coverage

was

extremely

strong

We

are

weak

in

the

coverage

of

systems

and

database

administration

We

will

continue

to

adjust

our

curriculum

as

IT

matures

as

an

academic

discipline

Information

Technology

is

maturing

rapidly

as

an

academic

discipline

A

public

draft

of

the

IT

volume

described

in

the

Computing

Curriculum

Overview

is

ready

for

review

The

SIGITE

Curriculum

Committee

is

soliciting

feedback

on

the

document

This

paper

presents

a

brief

history

of

SIGITE

the

ACM

SIG

for

Information

Technology

Education

and

a

brief

introduction

to

the

Information

Assurance

Education

community

The

authors

believe

that

collaboration

between

these

communities

can

be

of

benefit

to

all

of

the

participants

and

the

industry

at

large

SIGITE

and

the

CC

Joint

Task

Force

solicit

feedback

on

the

documents

at

The

authors

would

like

to

thank

the

ACM

Education

committee

for

their

support

of

the

effort

especially

Russ

Shackleford

without

whose

financial

support

and

encouragement

the

document

would

be

years

away

from

completion

We

would

also

like

to

express

appreciation

to

the

NSA

for

funding

the

IAEGC

-LSB-

-RSB-

program

Corey

Schou

's

IAEGC

lecture

on

helping

students

understand

IAS

in

an

hour

was

the

genesis

of

the

IAS

approach

in

The

BYU

authors

would

like

to

express

appreciation

to

our

colleagues

and

the

administration

of

the

School

of

Technology

at

Brigham

Young

University

who

covered

our

classes

and

found

the

funding

for

the

time

and

travel

our

participation

in

the

SIGITE

curriculum

committee

required

-LSB-

-RSB-

Ekstrom

Joseph

J.

Lunt

Barry

M.

Integration

of

Information

Assurance

and

Security

into

th

Colloquium

for

Information

Systems

Security

Education

June

Atlanta

Georgia

-LSB-

-RSB-

Lunt

Barry

M.

Ekstrom

Joseph

J.

Lawson

Edith

A.

Kamali

Reza

Miller

Jacob

Gorka

Sandra

Reichgelt

Han

Defining

the

IT

Curriculum

The

Results

of

the

Last

Years

World

Engineer

's

Convention

Shanghai

China

Nov

-LSB-

-RSB-

Joint

Task

Force

for

Computing

Curricula

-LRB-

-RRB-

Computing

Curricula

Computer

Science

Volume

December

Copyright

ACM/IEEE

-LSB-

-RSB-

-RSB-

Joint

Task

Force

for

Computing

Curricula

-LRB-

-RRB-

Computing

Curricula

Overview

Document

pdf

retrieved

Mar.

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

SIGITE

Curriculum

Committee

-LRB-

-RRB-

Computing

Curriculum

IT

Volume

-LSB-

-RSB-

NSA

web

site

Information

Assurance

Division

verified

Mar

-LSB-

-RSB-

Cerias

web

site

verified

Mar

-LSB-

-RSB-

Machonachy

W.

Victor

Schou

Corey

D.

Ragsdale

Daniel

Welch

Don

A

model

for

Information

Assurance

An

Integrated

Approach

Proceedings

of

the

IEEE

Workshop

on

Information

Assurance

and

Security

United

States

Military

Academy

West

Point

NY

June

-LSB-

-RSB-

Ekstrom

Joseph

Renshaw

Stephen

Curriculum

and

Issues

in

a

First

Course

of

Computer

Networking

for

Four-year

Information

Technology

Programs

ASEE

Session

-LSB-

-RSB-

Ekstrom

Joseph

Renshaw

Stephen

A

Project-Based

Introductory

Curriculum

in

Networking

WEB

and

Database

Systems

for

Information

Technology

Programs

CITC

Rochester

NY

September

-LSB-

-RSB-

Ekstrom

Joseph

Renshaw

Stephen

Database

Curriculum

Issues

for

Four-year

IT

Programs

CIEC

Tucson

AZ

January

-LSB-

-RSB-

Ekstrom

Joseph

Lunt

Barry

Education

at

the

Seams

Preparing

Students

to

Stitch

Systems

Together

Curriculum

and

Issues

for

IT

Programs

CITC

IV

Purdue

University

West

Lafayette

Indiana

October

-LSB-

-RSB-

Ekstrom

Joseph

Lunt

Barry

M

Helps

C.

Richard

Education

at

the

Seams

Preliminary

Evaluation

of

Teaching

Integration

as

a

Key

to

Education

in

Information

Technology

ASEE

Salt

Lake

City

Utah

June

-LSB-

-RSB-

Section

of

ref

-LSB-

-RSB-

-LSB-

-RSB-

IAEGC

Information

Assurance

Education

Graduate

Certificate

Validated

April

Appendix

A

From

Mar

Draft

The

Information

Technology

Body

of

Knowledge

ITF

Information

Technology

Fundamentals

-LRB-

core

-RRB-

Pervasive

Themes

in

IT

-LRB-

-RRB-

Organizational

Issues

-LRB-

-RRB-

History

of

IT

-LRB-

-RRB-

IT

and

Its

Related

and

Informing

Disciplines

-LRB-

-RRB-

Application

Domains

-LRB-

-RRB-

Applications

of

Math

and

Statistics

to

IT

-LRB-

-RRB-

HCI

Human

Computer

Interaction

-LRB-

core

hours

-RRB-

Human

Factors

-LRB-

-RRB-

HCI

Aspects

of

Application

Domains

-LRB-

-RRB-

Human-Centered

Evaluation

-LRB-

-RRB-

Developing

Effective

Interfaces

-LRB-

-RRB-

Accessibility

-LRB-

-RRB-

Emerging

Technologies

-LRB-

-RRB-

Human-Centered

Software

-LRB-

-RRB-

IAS

Information

Assurance

and

Security

-LRB-

core

-RRB-

Fundamental

Aspects

-LRB-

-RRB-

Security

Mechanisms

-LRB-

Countermeasures

-RRB-

-LRB-

-RRB-

Operational

Issues

-LRB-

-RRB-

Policy

-LRB-

-RRB-

Attacks

-LRB-

-RRB-

Security

Domains

-LRB-

-RRB-

Forensics

-LRB-

-RRB-

Information

States

-LRB-

-RRB-

Security

Services

-LRB-

-RRB-

Threat

Analysis

Model

-LRB-

-RRB-

Vulnerabilities

-LRB-

-RRB-

IM

Information

Management

-LRB-

core

hours

-RRB-

IM

Concepts

and

Fundamentals

-LRB-

-RRB-

Database

Query

Languages

-LRB-

-RRB-

Data

Organization

Architecture

-LRB-

-RRB-

Data

Modeling

-LRB-

-RRB-

Managing

the

Database

Environment

-LRB-

-RRB-

Special-Purpose

Databases

-LRB-

-RRB-

IPT

Integrative

Programming

Technologies

-LRB-

core

-RRB-

Intersystems

Communications

-LRB-

-RRB-

Data

Mapping

and

Exchange

-LRB-

-RRB-

Integrative

Coding

-LRB-

-RRB-

Scripting

Techniques

-LRB-

-RRB-

Software

Security

Practices

-LRB-

-RRB-

Miscellaneous

Issues

-LRB-

-RRB-

Overview

of

programming

languages

-LRB-

-RRB-

NET

Networking

-LRB-

core

hours

-RRB-

Foundations

of

Networking

-LRB-

-RRB-

Routing

and

Switching

-LRB-

-RRB-

Physical

Layer

-LRB-

-RRB-

Security

-LRB-

-RRB-

Application

Areas

-LRB-

-RRB-

Network

Management

PF

Programming

Fundamentals

-LRB-

core

hours

-RRB-

Fundamental

Data

Structures

-LRB-

-RRB-

Fundamental

Programming

Constructs

-LRB-

-RRB-

Object-Oriented

Programming

-LRB-

-RRB-

Algorithms

and

Problem-Solving

-LRB-

-RRB-

Event-Driven

Programming

-LRB-

-RRB-

Recursion

-LRB-

-RRB-

PT

Platform

Technologies

-LRB-

core

hours

-RRB-

Operating

Systems

-LRB-

-RRB-

Architecture

and

Organization

-LRB-

-RRB-

Computer

Infrastructure

-LRB-

-RRB-

Enterprise

Deployment

Software

Firmware

Hardware

SA

System

Administration

and

Maintenance

-LRB-

core

hours

-RRB-

Operating

Systems

-LRB-

-RRB-

Applications

-LRB-

-RRB-

Administrative

Activities

-LRB-

-RRB-

Administrative

Domains

-LRB-

-RRB-

SIA

System

Integration

and

Architecture

-LRB-

core

hours

-RRB-

Requirements

-LRB-

-RRB-

Acquisition/Sourcing

-LRB-

-RRB-

Integration

-LRB-

-RRB-

Project

Management

-LRB-

-RRB-

Testing

and

QA

-LRB-

-RRB-

Organizational

Context

-LRB-

-RRB-

Architecture

-LRB-

-RRB-

SP.

Social

and

Professional

Issues

-LRB-

core

hours

-RRB-

Technical

Writing

for

IT

-LRB-

-RRB-

History

of

Computing

-LRB-

-RRB-

Social

Context

of

Computing

-LRB-

-RRB-

Teamwork

Concepts

and

Issues

-LRB-

-RRB-

Intellectual

Properties

-LRB-

-RRB-

Legal

Issues

in

Computing

-LRB-

-RRB-

Organizational

Context

-LRB-

-RRB-

Professional

and

Ethical

Issues

and

Responsibilities

-LRB-

-RRB-

Privacy

and

Civil

Liberties

-LRB-

-RRB-

WS

Web

Systems

and

Technologies

-LRB-

core

hours

-RRB-

Web

Technologies

-LRB-

-RRB-

Information

Architecture

-LRB-

-RRB-

Digital

Media

-LRB-

-RRB-

Web

Development

-LRB-

-RRB-

Vulnerabilities

-LRB-

-RRB-

Social

Software

Total

Hours

Notes

Order

of

Knowledge

Areas

Fundamentals

first

then

ordered

alphabetically

Order

of

Units

under

each

Knowledge

Area

Fundamentals

first

-LRB-

if

present

-RRB-

then

ordered

by

number

of

core

hours

Appendix

B

O O B I O O O O B O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O B I O B I B O O O O O O B I O O B I B O O O O O O O O O O O O O O O B O O O O B O B O O B B O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O B I O O O O O B O O B O B O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O B O O O O O O O O O O O O O B I O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B O O B I O O O B O O O O O O O O O O O O B I B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B B O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O B O O O B I I O O O O O O O B O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O B I O O B O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O B I O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O B O O O O B I O O O O O B I O O O O O O O B I O B O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

byu curriculum
information technology
it2005
cc2005
training standards
security knowledge
information assurance
sigite curriculum committee
pervasive themes
it
education
ia
ias
committee on national security systems

in-service training development
nietp program
citc-1
it2005 volume


A Machine Learning Based Approach for Table Detection on The Web Table is a commonly used presentation scheme , especially for describing relational information . However , table understanding remains an open problem . In this paper , we consider the problem of table detection in web documents . Its potential applications include web mining , knowledge management , and web content summarization and delivery to narrow-bandwidth devices . We describe a machine learning based approach to classify each given table entity as either genuine or non-genuine . Various features re ecting the layout as well as content characteristics of tables are studied . In order to facilitate the training and evaluation of our table classi er , we designed a novel web document table ground truthing protocol and used it to build a large table ground truth database . The database consists of 1,393 HTML les collected from hundreds of di erent web sites and contains 11,477 leaf < TABLE > elements , out of which 1,740 are genuine tables . Experiments were conducted using the cross validation method and an F-measure of 95 : 89 % was achieved . H. 4.3 Information Systems Applications -RSB- : Communications Applications | Information browsers Algorithms The increasing ubiquity of the Internet has brought about a constantly increasing amount of online publications . As a compact and e cient way to present relational information , tables are used frequently in web documents . Since tables are inherently concise as well as information rich , the automatic understanding of tables has many applications including knowledge management , information retrieval , web Copyright is held by the author/owner -LRB- s -RRB- . WWW2002 , May 7 11 , 2002 , Honolulu , Hawaii , USA . ACM 1-58113-449-5 / 02/0005 . mining , summarization , and content delivery to mobile devices . The processes of table understanding in web documents include table detection , functional and structural analysis and nally table interpretation 6 -RSB- . In this paper , we concentrate on the problem of table detection . The web provides users with great possibilities to use their own style of communication and expressions . In particular , people use the < TABLE > tag not only for relational information display but also to create any type of multiple-column layout to facilitate easy viewing , thus the presence of the < TABLE > tag does not necessarily indicate the presence of a relational table . In this paper , we de ne genuine tables to be document entities where a two dimensional grid is semantically signi cant in conveying the logical relations among the cells 10 -RSB- . Conversely , Non-genuine tables are document entities where < TABLE > tags are used as a mechanism for grouping contents into clusters for easy viewing only . Figure 1 gives a few examples of genuine and non-genuine tables . While genuine tables in web documents could also be created without the use of < TABLE > tags at all , we do not consider such cases in this article as they seem very rare from our experience . Thus , in this study , Table detection refers to the technique which classi es a document entity enclosed by the < TABLE > < / TABLE > tags as genuine or non-genuine tables . Several researchers have reported their work on web table detection 2 , 10 , 6 , 14 -RSB- . In 2 -RSB- , Chen et al. used heuristic rules and cell similarities to identify tables . They tested their table detection algorithm on 918 tables from airline information web pages and achieved an F-measure of 86 : 50 % . Penn et al. proposed a set of rules for identifying genuinely tabular information and news links in HTML documents 10 -RSB- . They tested their algorithm on 75 web site front-pages and achieved an F-measure of 88 : 05 % . Yoshida et al. proposed a method to integrate WWW tables according to the category of objects presented in each table 14 -RSB- . Their data set contains 35,232 table tags gathered from the web . They estimated their algorithm parameters using all of table data and then evaluated algorithm accuracy on 175 of the tables . The average F-measure reported in their paper is 82 : 65 % . These previous methods all relied on heuristic rules and were only tested on a database that is either very small 10 -RSB- , or highly domain speci c 2 -RSB- . Hurst mentioned that a Naive Bayes classi er algorithm produced adequate results but no detailed algorithm and experimental information was provided 6 -RSB- . We propose a new machine learning based approach for 242 Figure 1 : Examples of genuine and non-genuine tables . table detection from generic web documents . In particular , we introduce a set of novel features which re ect the layout as well as content characteristics of tables . These features are used in classi ers trained on thousands of examples . To facilitate the training and evaluation of the table classi ers , we designed a novel web document table ground truthing protocol and used it to build a large table ground truth database . The database consists of 1,393 HTML les collected from hundreds of di erent web sites and contains 11,477 leaf < TABLE > elements , out of which 1,740 are genuine tables . Experiments on this database using the cross validation method demonstrate signi cant performance improvements over previous methods . The rest of the paper is organized as follows . We describe our feature set in Section 2 , followed by a brief discussion of the classi ers we experimented with in Section 3 . In Section 4 , we present a novel table ground truthing protocol and explain how we built our database . Experimental results are then reported in Section 5 and we conclude with future directions in Section 6 . Feature selection is a crucial step in any machine learning based methods . In our case , we need to nd a combination of features that together provide signi cant separation between genuine and non-genuine tables while at the same time constrain the total number of features to avoid the curse of dimensionality . Past research has clearly indicated that layout and content are two important aspects in table understanding 6 -RSB- . Our features were designed to capture both of these aspects . In particular , we developed 16 features which can be categorized into three groups : seven layout features , eight content type features and one word group feature . In the rst two groups , we attempt to capture the global composition of tables as well as the consistency within the whole table and across rows and columns . The last feature looks at words used in tables and is derived directly from the vector space model commonly used in Information Retrieval . Before feature extraction , each HTML document is rst parsed into a document hierarchy tree using Java Swing XML parser with W3C HTML 3.2 DTD 10 -RSB- . A < TABLE > node is said to be a leaf table if and only if there are no < TABLE > nodes among its children 10 -RSB- . Our experience indicates that almost all genuine tables are leaf tables . Thus in this study only leaf tables are considered candidates for genuine tables and are passed on to the feature extraction stage . In the following we describe each feature in detail . 2.1 Layout Features In HTML documents , although tags like < TR > and < TD > -LRB- or < TH > -RRB- may be assumed to delimit table rows and table cells , they are not always reliable indicators of the number of rows and columns in a table . Variations can be caused by spanning cells created using < ROWSPAN > and < COLSPAN > tags . Other tags such as < BR > could be used to move content into the next row . Therefore to extract layout features reliably one can not simply count the number of < TR > 's and < TD > 's . For this purpose , we maintain a matrix to record all 243 the cell spanning information and serve as a pseudo rendering of the table . Layout features based on row or column numbers are then computed from this matrix . Given a table T , assuming its numbers of rows and columns are rn and cn respectively , we compute the following layout features : Average number of columns , computed as the average number of cells per row : c = 1 rn rn X i = 1 c i where c i is the number of cells in row i , i = 1 : : : rn Standard deviation of number of columns : dC = v u u t 1 rn rn X i = 1 -LRB- c i ; c -RRB- -LRB- c i ; c -RRB- Average number of rows , computed as the average number of cells per column : r = 1 rn cn X i = 1 r i where r i is the number of cells in column i , i = 1 : : : cn Standard deviation of number of rows : dR = v u u t 1 cn cn X i = 1 -LRB- r i ; r -RRB- -LRB- r i ; r -RRB- : Since the majority of tables in web documents contain characters , we compute three more layout features based on cell length in terms of number of characters : Average overall cell length : cl = 1 en P en i = 1 cl i , where en is the total number of cells in a given table and cl i is the length of cell i , i = 1 : : : en Standard deviation of cell length : dCL = v u u t 1 en en X i = 1 -LRB- cl i ; cl -RRB- -LRB- cl i ; cl -RRB- Average Cumulative length consistency , CLC . The last feature is designed to measure the cell length consistency along either row or column directions . It is inspired by the fact that most genuine tables demonstrate certain consistency either along the row or the column direction , but usually not both , while non-genuine tables often show no consistency in either direction . First , the average cumulative within-row length consistency , CLC r , is computed as follows . Let the set of cell lengths of the cells from row i be R i , i = 1 : : : r -LRB- considering only non-spanning cells -RRB- : 1 . Compute the mean cell length , m i , for row R i . 2 . Compute cumulative length consistency within each R i : CLC i = X cl 2R i LC cl : Here LC cl is de ned as : LC cl = 0 : 5 ; D , where D = min fj cl ; m i j = m i 1 : 0 g . Intuitively , LC cl measures the degree of consistency between cl and the mean cell length , with ; 0 : 5 indicating extreme inconsistency and 0 : 5 indicating extreme consistency . When most cells within R i are consistent , the cumulative measure CLC i is positive , indicating a more or less consistent row . 3 . Take the average across all rows : CLC r = 1 r r X i = 1 CLC i : After the within-row length consistency CLC r is computed , the within-column length consistency CLC c is computed in a similar manner . Finally , the overall cumulative length consistency is computed as CLC = max -LRB- CLC r CLC c -RRB- . 2.2 Content Type Features Web documents are inherently multi-media and has more types of content than any traditional documents . For example , the content within a < TABLE > element could include hyperlinks , images , forms , alphabetical or numerical strings , etc. . Because of the relational information it needs to convey , a genuine table is more likely to contain alpha or numerical strings than , say , images . The content type feature was designed to re ect such characteristics . We de ne the set of content types T = f Image Form Hyperlink Alphabetical Digit Empty Others g . Our content type features include : The histogram of content type for a given table . This contributes 7 features to the feature set Average content type consistency , CTC . The last feature is similar to the cell length consistency feature . First , within-row content type consistency CTC r is computed as follows . Let the set of cell type of the cells from row i as T i , i = 1 : : : r -LRB- again , considering only non-spanning cells -RRB- : 1 . Find the dominant type , DT i , for T i . 2 . Compute the cumulative type consistency with each row R i , i = 1 : : : r : CTC i = X ct 2R i D where D = 1 if ct is equal to DT i and D = ; 1 , otherwise . 3 . Take the average across all rows : CTC r = 1 r r X i = 1 CTC i The within-column type consistency is then computed in a similar manner . Finally , the overall cumulative type consistency is computed as : CTC = max -LRB- CTC r CTC c -RRB- . 244 2.3 Word Group Feature If we treat each table as a \ mini-document '' by itself , table classi cation can be viewed as a document categorization problem with two broad categories : genuine tables and non-genuine tables . We designed the word group feature to incorporate word content for table classi cation based on techniques developed in information retrieval 7 , 13 -RSB- . After morphing 11 -RSB- and removing the infrequent words , we obtain the set of words found in the training data , W . We then construct weight vectors representing genuine and non-genuine tables and compare that against the frequency vector from each new incoming table . Let Z represent the non-negative integer set . The following functions are de ned on set W . df G : W ! Z , where df G -LRB- w i -RRB- is the number of genuine tables which include word w i , i = 1 : : : jW j tf G : W ! Z , where tf G -LRB- w i -RRB- is the number of times word w i , i = 1 : : : jW j , appears in genuine tables df N : W ! Z , where df N -LRB- w i -RRB- is the number of non-genuine tables which include word w i , i = 1 : : : jW j tf N : W ! Z , where tf N -LRB- w i -RRB- is the number of times word w i , i = 1 : : : jW j , appears in non-genuine tables . tf T : W ! Z , where tf T -LRB- w i -RRB- is the number of times word w i , w i 2 W appears in a new test table . To simplify the notations , in the following discussion , we will use df Gi , tf Gi , df Ni and tf Ni to represent df G -LRB- w i -RRB- , tf G -LRB- w i -RRB- , df N -LRB- w i -RRB- and tf N -LRB- w i -RRB- , respectively . Let N G , N N be the number of genuine tables and non-genuine tables in the training collection , respectively and let C = max -LRB- N G N N -RRB- . Without loss of generality , we assume N G 6 = 0 and N N 6 = 0 . For each word w i in W , i = 1 : : : jW j , two weights , p G i and p N i are computed : p G i = 8 < : tf Gi log -LRB- df G i N G N N df N i + 1 -RRB- when df Ni 6 = 0 tf Gi log -LRB- df G i N G C + 1 -RRB- when df Ni = 0 p N i = 8 < : tf Ni log -LRB- df N i N N N G df G i + 1 -RRB- when df Gi 6 = 0 tf Ni log -LRB- df N i N N C + 1 -RRB- when df Gi = 0 As can be seen from the formulas , the de nitions of these weights were derived from the traditional tf idf measures used in informational retrieval , with some adjustments made for the particular problem at hand . Given a new incoming table , let us denote the set including all the words in it as W n . Since W is constructed using thousands of tables , the words that are present in both W and W n are only a small subset of W . Based on the vector space model , we de ne the similarity between weight vectors representing genuine and non-genuine tables and the frequency vector representing the incoming table as the corresponding dot products . Since we only need to consider the words that are present in both W and W n , we rst compute the e ective word set : W e = W \ W n . Let the words in W e be represented as w m k , where m k k = 1 : : : jW e j , are indexes to the words from set W = f w 1 w 2 : : : w jW j g . we de ne the following vectors : Weight vector representing the genuine table group : ! G S = p G m 1 U p G m 2 U p G m jW e j U ! where U is the cosine normalization term : U = v u u t jW e j X k = 1 p G m k p G m k : Weight vector representing the non-genuine table group : ! N S = p N m 1 V p N m 2 V p N m jW e j V ! where V is the cosine normalization term : V = v u u t jW e j X k = 1 p N m k p N m k : Frequency vector representing the new incoming table : ! I T = tf Tm 1 tf Tm 2 tf Tm jW e j : Finally , the word group feature is de ned as the ratio of the two dot products : wg = 8 > > > < > > > : ! I T ! G S ! I T ! N S when ! I T ! N S 6 = 0 1 when ! I T ! G S = 0 and ! I T ! N S = 0 10 when ! I T ! G S 6 = 0 and ! I T ! N S = 0 Various classi cation schemes have been widely used in document categorization as well as web information retrieval 13 , 8 -RSB- . For the table detection task , the decision tree classi-er is particularly attractive as our features are highly non-homogeneous . We also experimented with Support Vector Machines -LRB- SVM -RRB- , a relatively new learning approach which has achieved one of the best performances in text categorization 13 -RSB- . 3.1 Decision Tree Decision tree learning is one of the most widely used and practical methods for inductive inference . It is a method for approximating discrete-valued functions that is robust to noisy data . Decision trees classify an instance by sorting it down the tree from the root to some leaf node , which provides the classi cation of the instance . Each node in a discrete-valued decision tree speci es a test of some attribute of the instance , and each branch descending from that node corresponds to one of the possible values for this attribute . Continuous-valued decision attributes can be incorporated by dynami-cally de ning new discrete-valued attributes that partition the continuous attribute value into a discrete set of intervals 9 -RSB- . Animplementationof thecontinuous-valueddecision tree described in 4 -RSB- was used for our experiments . The decision tree is constructed using a training set of feature vectors with true class labels . At each node , a discriminant threshold 245 is chosen such that it minimizes an impurity value . The learned discriminant function splits the training subset into two subsets and generates two child nodes . The process is repeated at each newly generated child node until a stopping condition is satis ed , and the node is declared as a terminal node based on a majority vote . The maximum impurity reduction , the maximum depth of the tree , and minimum number of samples are used as stopping conditions . 3.2 SVM Support Vector Machines -LRB- SVM -RRB- are based on the Structural Risk Management principle from computational learning theory 12 -RSB- . The idea of structural risk minimization is to nd a hypothesis h for which the lowest true error is guaranteed . The true error of h is the probability that h will make an error on an unseen and randomly selected test example . The SVM method is de ned over a vector space where the goal is to nd a decision surface that best separates the data points in two classes . More precisely , the decision surface by SVM for linearly separable space is a hyperplane which can be written as ~ w ~ x ; b = 0 where ~ x is an arbitrary data point and the vector ~ w and the constant b are learned from training data . Let D = -LRB- y i ~ x i -RRB- denote the training set , and y i 2 f +1 ; 1 g be the classi cation for ~ x i , the SVM problem is to nd ~ w and b that satis es the following constraints : ~ w ~ x i ; b +1 for y i = +1 ~ w ~ x i ; b ; 1 for y i = ; 1 while minimizing the vector 2-norm of ~ w . The SVM problem in linearly separable cases can be e ciently solved using quadratic programming techniques , while the non-linearly separable cases can be solved by either introducing soft margin hyperplanes , or by mapping the original data vectors to a higher dimensional space where the data points become linearly separable 12 , 3 -RSB- . One reason why SVMs are very powerful is that they are very universal learners . In their basic form , SVMs learn linear threshold functions . Nevertheless , by a simple \ plug-in '' of an appropriate kernel function , they can be used to learn polynomial classi ers , radial basis function -LRB- RBF -RRB- networks , three-layer sigmoid neural nets , etc. 3 -RSB- . For our experiments , we used the SV M light system implemented by Thorsten Joachims . 1 Since there are no publicly available web table ground truth database , researchers tested their algorithms in di erent data sets in the past 2 , 10 , 14 -RSB- . However , their data sets either had limited manually annotated table data -LRB- e.g. , 918 table tags in 2 -RSB- , 75 HTML pages in 10 -RSB- , 175 manually annotated table tags in 14 -RSB- -RRB- , or were collected from some speci c domains -LRB- e.g. , a set of tables selected from airline information pages were used in 2 -RSB- -RRB- . To develop our machine learning based table detection algorithm , we needed to build a general web table ground truth database of signi cant size . 1 http://svmlight.joachims.org 4.1 Data Collection Instead of working within a speci c domain , our goal of data collection was to get tables of as many di erent varieties as possible from the web . To accomplish this , we composed a set of key words likely to indicate documents containing tables and used those key words to retrieve and download web pages using the Google search engine . Three directories on Google were searched : the business directory and news directory using key words : ftable , stock , bonds , figure , schedule , weather , score , service , results , valueg , and the science directory using key words ftable , results , valueg . A total of 2,851 web pages were down-loaded in this manner and we ground truthed 1,393 HTML pages out of these -LRB- chosen randomly among all the HTML pages -RRB- . These 1,393 HTML pages from around 200 web sites comprise our database . 4.2 Ground Truthing There has been no previous report on how to systemati-cally generate web table ground truth data . To build a large web table ground truth database , a simple , exible and complete ground truth protocol is required . Figure 4.2 -LRB- a -RRB- shows the diagram of our ground truthing procedure . We created a new Document Type De nition -LRB- DTD -RRB- which is a super-set of W3C HTML 3.2 DTD . We added three attributes for < TABLE > element , which are \ tabid '' , \ genuine table '' and \ table title '' . The possible value of the second attribute is yes or no and the value of the rst and third attributes is a string . We used these three attributes to record the ground truth of each leaf < TABLE > node . The bene t of this design is that the ground truth data is inside HTML le format . We can use exactly the same parser to process the ground truth data . We developed a graphical user interface for web table ground truthing using the Java 1 -RSB- language . Figure 4.2 -LRB- b -RRB- is a snapshot of the interface . There are two windows . After reading an HTML le , the hierarchy of the HTML le is shown in the left window . When an item is selected in the hierarchy , the HTML source for the selected item is shown in the right window . There is a panel below the menu bar . The user can use the radio button to select either genuine table or non-genuine table . The text window is used to input table title . 4.3 Database Description Our nal table ground truth database consists of 1,393 HTML pages collected from around 200 web sites . There are a total of 14,609 < TABLE > nodes , including 11,477 leaf < TABLE > nodes . Out of the 11,477 leaf < TABLE > nodes , 1,740 are genuine tables and 9,737 are non-genuine tables . Not every genuine table has its title and only 1,308 genuine tables have table titles . We also found at least 253 HTML les have unmatched < TABLE > , < / TABLE > pairs or wrong hierarchy , which demonstrates the noisy nature of web documents A hold-out method is used to evaluate our table classi-er . We randomly divided the data set into nine parts . Each classi er was trained on eight parts and then tested on the remaining one part . This procedure was repeated nine times , each time with a di erent choice for the test 246 Parser Adding attributes HTML with attributes and unique index to each table -LRB- ground truth -RRB- Validation HTML File Hierarchy -LRB- a -RRB- -LRB- b -RRB- Figure 2 : -LRB- a -RRB- The diagram of ground truthing procedure -LRB- b -RRB- A snapshot of the ground truthing software . part . Then the combined nine part results are averaged to arrive at the overall performance measures 4 -RSB- . For the layout and content type features , this procedure is straightforward . However it is more complicated for the word group feature training . To compute w g for training samples , we need to further divide the training set into two groups , a larger one -LRB- 7 parts -RRB- for the computation of the weights p G i and p N i , i = 1 : : : jW j , and a smaller one -LRB- 1 part -RRB- for the computation of the vectors ! G S , ! N S , and ! I T . This partition is again rotated to compute w g for each table in the training set . Table 1 : Possible true - and detected-state combinations for two classes . True Class Assigned Class genuine table non-genuine table genuine table N gg N gn non-genuine table N ng N nn The output of each classi er is compared with the ground truth and a contingency table is computed to indicate the number of a particular class label that are identi ed as members of one of two classes . The rows of the contingency table represent the true classes and the columns represent the assigned classes . The cell at row r and column c is the number of tables whose true class is r while its assigned class is c . The possible true - and detected-state combination is shown in Table 1 . Three performance measures Recall Rate -LRB- R -RRB- , Precision Rate -LRB- P -RRB- and F-measure -LRB- F -RRB- are computed as follows : R = N gg N gg + N gn P = N gg N gg + N ng F = R + P 2 : For comparison among di erent features and learning algorithms we report the performance measures when the best F-measure is achieved . First , the performance of various feature groups and their combinations were evaluated using the decision tree classi er . The results are given in Table 2 . Table 2 : Experimental results using various feature groups and the decision tree classi er . L T LT LTW R -LRB- % -RRB- 87.24 90.80 94.20 94.25 P -LRB- % -RRB- 88.15 95.70 97.27 97.50 F -LRB- % -RRB- 87.70 93.25 95.73 95.88 L : La y out only . T : Con ten t t yp e only . L T : La y out and con ten t t yp e. L TW : La y out , con ten t t yp e and w ord group . As seen from the table , content type features performed better than layout features as a single group , achieving an F-measure of 93 : 25 % . However , when the two groups were combined the F-measure was improved substantially to 95 : 73 % , recon rming the importance of combining layout and content features in table detection . The addition of the word group feature improved the F-measure slightly more to 95 : 88 % . Table 3 compares the performances of di erent learning algorithms using the full feature set . The leaning algorithms tested include the decision tree classi er and the SVM al-247 gorithm with two di erent kernels -LCB- linear and radial basis function -LRB- RBF -RRB- . Table 3 : Experimental results using di erent learning algorithms . Tree SVM -LRB- linear -RRB- SVM -LRB- RBF -RRB- R -LRB- % -RRB- 94.25 93.91 95.98 P -LRB- % -RRB- 97.50 91.39 95.81 F -LRB- % -RRB- 95.88 92.65 95.89 As seen from the table , for this application the SVM with radial basis function kernel performed much better than the one with linear kernel . It achieved an F measure of 95 : 89 % , comparable to the 95 : 88 % achieved by the decision tree classi er . Figure 3 shows two examples of correctly classi ed tables , where Figure 3 -LRB- a -RRB- is a genuine table and Figure 3 -LRB- b -RRB- is a non-genuine table . Figure 4 shows a few examples where our algorithm failed . Figure 4 -LRB- a -RRB- was misclassi ed as a non-genuine table , likely because its cell lengths are highly inconsistent and it has many hyperlinks which is unusual for genuine tables . The reason why Figure 4 -LRB- b -RRB- was misclassi ed as non-genuine is more interesting . When we looked at its HTML source code , we found it contains only two < TR > tags . All text strings in one rectangular box are within one < TD > tag . Its author used < p > tags to put them in di erent rows . This points to the need for a more carefully designed pseudo-rendering process . Figure 4 -LRB- c -RRB- shows a non-genuine table misclassi-ed as genuine . A close examination reveals that it indeed has good consistency along the row direction . In fact , one could even argue that this is indeed a genuine table , with implicit row headers of Title , Name , Company A liation and Phone Number . This example demonstrates one of the most di cult challenges in table understanding , namely the ambiguous nature of many table instances -LRB- see 5 -RSB- for a more detailed analysis on that -RRB- . Figure 4 -LRB- d -RRB- was also misclassi-ed as a genuine table . This is a case where layout features and the kind of shallow content features we used are not enough | deeper semantic analysis would be needed in order to identify the lack of logical coherence which makes it a non-genuine table . For comparison , we tested the previously developed rule-based system 10 -RSB- on the same database . The initial results -LRB- shown in Table 4 under \ Original Rule Based '' -RRB- were very poor . After carefully studying the results from the initial experiment we realized that most of the errors were caused by a rule imposing a hard limit on cell lengths in genuine tables . After deleting that rule the rule-based system achieved much improved results -LRB- shown in Table 4 under \ Modi ed Rule Based '' -RRB- . However , the proposed machine learning based method still performs considerably better in comparison . This demonstrates that systems based on handcrafted rules tend to be brittle and do not generalize well . In this case , even after careful manual adjustment in a new database , it still does not work as well as an automatically trained classi er . -LRB- a -RRB- -LRB- b -RRB- Figure 3 : Examples of correctly classi ed tables . -LRB- a -RRB- : a genuine table -LRB- b -RRB- : a non-genuine table . Table 4 : Experimental results of a previously developed rule based system . Original Rule Based Modi ed Rule Based R -LRB- % -RRB- 48.16 95.80 P -LRB- % -RRB- 75.70 79.46 F -LRB- % -RRB- 61.93 87.63 248 -LRB- a -RRB- -LRB- b -RRB- -LRB- c -RRB- -LRB- d -RRB- Figure 4 : Examples of misclassi ed tables . -LRB- a -RRB- and -LRB- b -RRB- : Genuine tables misclassi ed as non-genuine -LRB- c -RRB- and -LRB- d -RRB- : Non-genuine tables misclassi ed as genuine . A direct comparison to other previous results 2 , 14 -RSB- is not possible currently because of the lack of access to their system . However , our test database is clearly more general and far larger than the ones used in 2 -RSB- and 14 -RSB- , while our precision and recall rates are both higher . Table detection in web documents is an interesting and challenging problem with many applications . We present a machine learning based table detection algorithm for HTML documents . Layout features , content type features and word group features were used to construct a novel feature set . Decision tree and SVM classi ers were then implemented and tested in this feature space . We also designed a novel table ground truthing protocol and used it to construct a large web table ground truth database for training and testing . Experiments on this large database yielded very promising results . Our future work includes handling more di erent HTML styles in pseudo-rendering , detecting table titles of the rec-ognized genuine tables and developing a machine learning based table interpretation algorithm . We would also like to investigate ways to incorporate deeper language analysis for both table detection and interpretation . We would like to thank Kathie Shipley for her help in collecting the web pages , and Amit Bagga for discussions on vector space models . 1 -RSB- M. Campione , K. Walrath , and A. Huml . The java -LRB- tm -RRB- tutorial : A short course on the basics -LRB- the java -LRB- tm -RRB- series -RRB- . 2 -RSB- H.-H . Chen , S.-C . Tsai , and J.-H . Tsai . Mining tables from large scale html texts . In Proc . 18th International Conference on Computational Linguistics , Saabrucken , Germany , July 2000 . 3 -RSB- C. Cortes and V. Vapnik . Support-vector networks . Machine Learning , 20:273 -LCB- 296 , August 1995 . 4 -RSB- R. Haralick and L. Shapiro . Computer and Robot Vision , volume 1 . Addison Wesley , 1992 . 5 -RSB- J. Hu , R. Kashi , D. Lopresti , G. Nagy , and G. Wilfong . Why table ground-truthing is hard . In Proc . 6th International Conference on Document Analysis and Recognition -LRB- ICDAR01 -RRB- , pages 129 -LCB- 133 , Seattle , WA , USA , September 2001 . 6 -RSB- M. Hurst . Layout and language : Challenges for table understanding on the web . In Proc . 1st International Workshop on Web Document Analysis , pages 27 -LCB- 30 , Seattle , WA , USA , September 2001 . 7 -RSB- T. Joachims . A probabilistic analysis of the rocchio algorithm with t df for text categorization . In Proc . 14th International Conference on Machine Learning , pages 143 -LCB- 151 , Morgan Kaufmann , 1997 . 8 -RSB- A. McCallum , K. Nigam , J. Rennie , and K. Seymore . Automating the construction of internet portals with machine learning . In Information Retrieval Journal , volume 3 , pages 127 -LCB- 163 , Kluwer , 2000 . 249 9 -RSB- T. M. Mitchell . Machine Learning . McGraw-Hill , 1997 . 10 -RSB- G. Penn , J. Hu , H. Luo , and R. McDonald . Flexible web document analysis for delivery to narrow-bandwidth devices . In Proc . 6th International Conference on Document Analysis and Recognition -LRB- ICDAR01 -RRB- , pages 1074 -LCB- 1078 , Seattle , WA , USA , September 2001 . 11 -RSB- M. F. Porter . An algorithm for su x stripping . Program , 14 -LRB- 3 -RRB- : 130 -LCB- 137 , 1980 . 12 -RSB- V. N. Vapnik . The Nature of Statistical Learning Theory , volume 1 . Springer , New York , 1995 . 13 -RSB- Y. Yang and X. Liu . A re-examination of text categorization methods . In Proc . SIGIR '99 , pages 42 -LCB- 49 , Berkeley , California , USA , August 1999 . 14 -RSB- M. Yoshida , K. Torisawa , and J. Tsujii . A method to integrate tables of the world wide web . In Proc . 1st International Workshop on Web Document Analysis , pages 31 -LCB- 34 , Seattle , WA , USA , September 2001 . 250

O B I I I O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O B I O O O O O B O O B I O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

information retrieval
word group
table ground truthing protocol
presentation
machine learning based approach
html document
decision tree
table detection
content type
machine learning
algorithms
layout

support vector machine
layout analysis
classifers
classifcation schemes