Breadth-First

Search

Crawling

Yields

High-Quality

Pages

This

paper

examines

the

average

page

quality

over

time

of

pages

downloaded

during

a

web

crawl

of

million

unique

pages

We

use

the

connectivity-based

metric

PageRank

to

measure

the

quality

of

a

page

We

show

that

traversing

the

web

graph

in

breadth-first

search

order

is

a

good

crawling

strategy

as

it

tends

to

discover

high-quality

pages

early

on

in

the

crawl

According

to

a

study

released

in

October

the

directly

accessible

surface

web

consists

of

about

billion

pages

while

the

deep

web

-LRB-

dynamically

generated

web

pages

-RRB-

consists

of

about

billion

pages

of

which

are

publicly

accessible

-LSB-

-RSB-

By

comparison

the

Google

index

released

in

June

contained

million

full-text-indexed

pages

-LSB-

-RSB-

In

other

words

Google

which

according

to

a

recent

measurement

-LSB-

-RSB-

has

the

greatest

coverage

of

all

search

engines

covers

only

about

of

the

publicly

accessible

web

and

the

other

major

search

engines

do

even

worse

Increasing

the

coverage

of

existing

search

engines

by

three

orders

of

magnitude

would

pose

a

number

of

technical

challenges

both

with

respect

to

their

ability

to

discover

download

and

index

web

pages

as

well

as

their

ability

to

serve

queries

against

an

index

of

that

size

-LRB-

For

query

engines

based

on

inverted

lists

the

cost

of

serving

a

query

is

linear

to

the

size

of

the

index

-RRB-

Therefore

search

engines

should

attempt

to

download

the

best

pages

and

include

-LRB-

only

-RRB-

them

in

their

index

Cho

Garcia-Molina

and

Page

-LSB-

-RSB-

suggested

using

connectivity

based

document

quality

metrics

to

direct

a

crawler

towards

high-quality

pages

They

performed

a

series

of

crawls

over

pages

in

the

stanford.edu

domain

and

used

Copyright

is

held

by

the

author/owner

May

Hong

Kong

ACM

different

ordering

metrics

breadth-first

backlink

count

PageRank

-LSB-

-RSB-

and

random

to

direct

the

different

crawls

Under

the

breath-first

ordering

pages

are

crawled

in

the

order

they

are

discovered

Under

the

backlink

ordering

the

pages

with

the

highest

number

of

known

links

to

them

are

crawled

first

Under

the

PageRank

ordering

pages

with

the

highest

PageRank

-LRB-

a

page

quality

metric

described

below

-RRB-

are

crawled

first

Under

the

random

ordering

the

crawler

selects

the

next

page

to

download

at

random

from

the

set

of

uncrawled

pages

-LRB-

For

repeatability

these

crawls

were

virtual

that

is

they

were

performed

over

a

cached

copy

of

these

pages

-RRB-

Cho

et

al.

evaluated

the

effectiveness

of

each

ordering

metric

by

examining

how

fast

it

led

the

crawler

to

all

the

hot

pages

In

this

context

a

hot

page

is

a

page

with

either

a

high

number

of

links

pointing

to

it

or

a

page

with

a

high

PageRank

They

found

that

using

the

PageRank

metric

to

direct

a

crawler

works

extremely

well

However

they

also

discovered

that

performing

the

crawl

in

breadth-first

order

works

almost

as

well

in

particular

if

hot

pages

are

defined

to

be

pages

with

high

PageRank

This

paper

extends

the

results

of

Cho

et

al.

regarding

the

effectiveness

of

crawling

in

breadth-first

search

order

using

a

much

larger

and

more

diverse

data

set

While

Cho

's

work

was

based

on

a

crawl

of

pages

from

the

stanford.edu

domain

we

performed

a

crawl

of

million

pages

over

the

entire

web

covering

more

than

million

distinct

hosts

We

use

connectivity-based

page

quality

metrics

namely

Brin

and

Page

's

PageRank

and

variations

of

it

to

measure

the

quality

of

downloaded

pages

over

the

life

of

the

crawl

We

find

that

not

only

does

breadth-first

search

download

the

hot

pages

first

but

also

that

the

average

quality

of

the

pages

decreased

over

the

duration

of

the

crawl

We

also

suggest

that

our

crawler

's

modifications

to

strict

breadth-first

search

made

to

increase

the

overall

download

rate

and

to

avoid

overloading

any

given

web

server

enhance

its

likeliness

of

retrieving

important

pages

first

The

remainder

of

this

paper

is

structured

as

follows

Section

reviews

the

PageRank

metric

we

used

to

evaluate

the

effectiveness

of

crawling

in

breadth-first

search

order

Section

describes

the

tools

we

used

to

conduct

our

experiments

Section

describes

the

experiments

we

performed

and

the

results

we

obtained

Finally

section

offers

concluding

remarks

There

are

many

conceivable

metrics

for

judging

the

quality

of

a

web

page

by

analyzing

its

content

by

measuring

its

popularity

-LRB-

that

is

how

often

it

is

viewed

-RRB-

or

by

examining

its

connectivity

-LRB-

that

is

by

determining

which

other

pages

link

to

this

page

and

vice

versa

-RRB-

Metrics

based

on

connectivity

have

the

advantages

that

they

do

not

require

information

that

is

not

easily

accessible

-LRB-

such

as

page

popularity

data

-RRB-

and

that

they

are

easy

to

compute

so

they

scale

well

to

even

very

large

page

collections

They

also

require

retrieving

only

the

links

on

each

page

not

the

full

page

contents

Storing

the

full

page

contents

requires

several

kilobytes

per

page

one

to

two

orders

of

magnitude

more

than

just

storing

the

links

PageRank

is

the

connectivity-based

page

quality

measure

suggested

by

Brin

and

Page

-LSB-

-RSB-

It

is

a

static

measure

it

is

designed

to

rank

pages

in

the

absence

of

any

queries

That

is

PageRank

computes

the

global

worth

of

each

page

Intuitively

the

PageRank

measure

of

a

page

is

similar

to

its

in-degree

which

is

a

possible

measure

of

the

importance

of

a

page

The

PageRank

of

a

page

is

high

if

many

pages

with

a

high

PageRank

contain

links

to

it

and

a

page

containing

few

outgoing

links

contributes

more

weight

to

the

pages

it

links

to

than

a

page

containing

many

outgoing

links

The

PageRank

of

a

page

is

expressed

mathematically

as

follows

Suppose

there

are

T

total

pages

on

the

web

We

choose

a

parameter

d

-LRB-

explained

below

-RRB-

such

that

d

a

typical

value

of

d

might

lie

in

the

range

d

Let

pages

p

p

p

k

link

to

page

p.

Let

R

-LRB-

p

i

-RRB-

be

the

PageRank

of

p

i

and

C

-LRB-

p

i

-RRB-

be

the

number

of

links

out

of

p

i

Then

the

PageRank

R

-LRB-

p

-RRB-

of

page

p

is

defined

to

satisfy

R

-LRB-

p

-RRB-

d

T

-LRB-

d

-RRB-

k

X

i

R

-LRB-

p

i

-RRB-

C

-LRB-

p

i

-RRB-

This

equation

defines

R

-LRB-

p

-RRB-

uniquely

modulo

a

constant

scaling

factor

If

we

scale

R

-LRB-

p

-RRB-

so

that

the

PageRanks

of

all

pages

sum

to

R

-LRB-

p

-RRB-

can

be

thought

of

as

a

probability

distribution

over

pages

The

PageRank

distribution

has

a

simple

interpretation

in

terms

of

a

random

walk

Imagine

a

web

surfer

who

wanders

the

web

If

the

surfer

visits

page

p

the

random

walk

is

in

state

p

At

each

step

the

web

surfer

either

jumps

to

a

page

on

the

web

chosen

uniformly

at

random

or

the

web

surfer

follows

a

link

chosen

uniformly

at

random

from

those

on

the

current

page

The

former

occurs

with

probability

d

the

latter

with

probability

d

The

equilibrium

probability

that

such

a

surfer

is

at

page

p

is

simply

R

-LRB-

p

-RRB-

An

alternative

way

to

say

this

is

that

the

average

fraction

of

the

steps

that

a

walk

spends

at

page

p

is

R

-LRB-

p

-RRB-

over

sufficiently

long

walks

This

means

that

pages

with

high

PageRank

are

more

likely

to

be

visited

than

pages

with

low

PageRank

In

our

experiments

we

set

d

We

also

modified

PageRank

slightly

so

that

pages

with

no

outgoing

links

contribute

their

weight

equally

to

all

pages

That

is

the

random

surfer

is

equally

likely

to

jump

to

any

page

from

a

page

with

no

outgoing

links

We

ran

experiments

using

both

the

original

PageRank

algorithm

which

does

not

distinguish

between

links

to

pages

on

the

same

versus

different

hosts

and

a

variant

of

PageRank

which

only

considers

links

to

different

hosts

We

used

two

tools

in

conducting

this

research

Mercator

and

the

Connectivity

Server

both

developed

at

our

lab

We

used

Mercator

to

crawl

the

web

and

the

Connectivity

Server

to

provide

fast

access

to

the

link

information

downloaded

from

the

crawl

Mercator

is

an

extensible

multithreaded

high-performance

web

crawler

-LSB-

-RSB-

It

is

written

in

Java

and

is

highly

configurable

Its

default

download

strategy

is

to

perform

a

breadth-first

search

of

the

web

with

the

following

three

modifications

It

downloads

multiple

pages

-LRB-

typically

-RRB-

in

parallel

This

modification

allows

us

to

download

about

million

pages

a

day

without

it

we

would

download

well

under

pages

per

day

Only

a

single

HTTP

connection

is

opened

to

any

given

web

server

at

any

given

time

This

modification

is

necessary

due

to

the

prevalence

of

relative

URLs

on

the

web

-LRB-

about

of

the

links

on

an

average

web

page

refer

to

the

same

host

-RRB-

which

leads

to

a

high

degree

of

host

locality

in

the

crawler

's

download

queue

If

we

were

to

download

many

pages

from

the

same

host

in

parallel

we

would

overload

or

even

crash

that

web

server

If

it

took

t

seconds

to

download

a

document

from

a

given

web

server

then

Mercator

will

wait

for

seconds

before

contacting

that

web

server

again

This

modification

is

not

strictly

necessary

but

it

further

eases

the

load

our

crawler

places

on

individual

servers

on

the

web

We

found

that

this

policy

reduces

the

rate

of

complaints

we

receive

while

crawling

For

the

experiments

described

below

we

configured

Mercator

to

extract

all

the

links

from

each

downloaded

page

and

save

them

to

disk

for

disk

space

reasons

we

did

not

retain

the

pages

themselves

We

conducted

a

crawl

that

attempted

to

download

million

pages

over

the

course

of

days

-LRB-

which

we

refer

to

as

days

to

throughout

the

paper

-RRB-

Of

all

those

download

attempts

million

returned

valid

unique

HTML

pages

the

others

resulted

in

TCP

and

DNS-errors

HTTP

return

codes

non-HTML

documents

or

duplicates

Mercator

's

download

rate

decreased

over

the

course

of

the

crawl

due

to

increasing

access

times

to

one

of

its

disk-based

data

structures

that

keeps

track

of

which

URLs

have

already

been

seen

The

median

download

day

was

the

mean

download

day

was

The

extracted

links

data

was

then

loaded

into

the

Connectivity

Server

-LRB-

-RRB-

-LSB-

-RSB-

a

database

for

URLs

and

links

A

build

of

takes

a

web

crawl

as

input

and

creates

a

database

representation

of

the

web

graph

induced

by

the

pages

in

the

crawl

A

database

consists

of

all

URLs

that

were

crawled

extended

with

all

URLs

referenced

at

least

five

times

by

the

crawled

pages

-LRB-

Incorporating

uncrawled

URLs

with

multiple

links

pointing

to

them

ensured

that

we

did

not

ignore

any

popular

URLs

Setting

the

threshold

at

five

incoming

links

reduced

the

set

of

uncrawled

URLs

by

over

which

enabled

us

to

fit

the

database

within

the

GB

of

RAM

available

to

us

-RRB-

The

database

also

contains

all

links

among

those

URLs

and

host

information

for

each

URL

It

maps

each

URL

to

all

of

its

outgoing

and

its

incoming

links

It

is

possible

to

get

all

the

incoming

links

for

a

given

URL

or

just

the

links

from

different

hosts

stores

links

in

both

directions

in

on

average

bytes

per

link

-LRB-

as

compared

to

bytes

per

link

in

the

original

connectivity

server

-LRB-

-RRB-

described

in

-LSB-

-RSB-

-RRB-

Like

Day

of

crawl

Average

PageRank

Figure

Average

PageRank

score

by

day

of

crawl

is

designed

to

give

high-performance

access

when

run

on

a

machine

with

enough

RAM

to

store

the

database

in

memory

On

the

MHz

Compaq

AlphaServer

with

GB

of

RAM

used

in

our

experiments

it

takes

ms

to

convert

a

URL

into

an

internal

id

or

vice

versa

and

ms/link

to

retrieve

each

incoming

or

outgoing

link

as

an

internal

id

The

database

for

our

crawl

of

million

pages

contained

million

URLs

and

billion

links

Therefore

one

iteration

of

PageRank

ran

in

about

minutes

In

this

section

we

report

on

our

experiments

We

implemented

PageRank

and

its

variants

over

the

interface

and

ran

each

algorithm

for

iterations

on

the

billion

link

database

-LRB-

In

all

our

experiments

the

PageRank

computation

converged

within

less

than

iterations

-RRB-

Although

the

PageRank

scores

are

conventionally

normalized

to

sum

to

-LRB-

making

it

easier

to

think

of

them

as

a

probability

distribution

-RRB-

we

normalized

them

to

sum

to

the

number

of

nodes

in

the

graph

-LRB-

million

-RRB-

This

way

the

average

page

has

a

PageRank

of

independent

of

the

number

of

pages

Figure

shows

the

average

PageRank

of

all

pages

downloaded

on

each

day

of

the

crawl

The

average

score

for

pages

crawled

on

the

first

day

is

more

than

three

times

the

average

score

of

for

pages

crawled

on

the

second

day

The

average

score

tapers

from

there

down

to

after

the

first

week

after

the

second

week

and

after

the

fourth

week

Clearly

we

downloaded

more

high

quality

pages

i.e.

pages

with

high

PageRank

early

in

the

crawl

than

later

on

We

then

decided

to

examine

specifically

when

we

had

crawled

the

highest

ranked

pages

We

examined

the

pages

with

the

top

N

PageRanks

for

increasing

values

of

N

from

to

million

-LRB-

all

of

the

pages

downloaded

-RRB-

Figure

graphs

the

average

day

on

which

we

crawled

the

pages

with

the

highest

N

scores

Note

that

the

horizontal

axis

shows

the

values

of

N

on

a

log

scale

All

of

the

top

and

of

the

top

pages

were

crawled

on

the

first

day

There

are

some

anomalies

in

the

graph

between

N

equals

and

where

the

average

day

fluctuates

between

and

-LRB-

the

second

and

third

days

of

the

crawl

-RRB-

These

anomalies

are

caused

by

pages

in

the

top

-LRB-

-RRB-

that

were

downloaded

after

the

first

week

Most

of

those

pages

had

a

lot

of

local

links

-LRB-

links

from

pages

on

the

same

host

-RRB-

but

not

many

remote

links

In

other

words

the

top

N

Average

day

top

N

pages

were

crawled

Figure

Average

day

on

which

the

top

N

pages

were

crawled

pages

on

the

same

host

endorse

each

other

but

few

other

hosts

endorse

them

We

address

this

phenomenon

later

in

the

last

experiment

shown

in

Figure

After

N

equals

the

curve

steadily

increases

to

day

the

mean

download

day

of

the

entire

crawl

Our

next

experiment

checks

that

pages

with

high

PageRank

are

not

ranked

high

only

because

they

were

crawled

early

For

example

a

page

whose

outgoing

links

all

point

to

pages

with

links

back

to

it

might

have

an

artificially

high

PageRank

if

all

of

its

outgoing

links

have

been

crawled

but

not

too

many

other

pages

For

this

experiment

we

ran

the

PageRank

algorithm

on

the

graph

induced

by

only

the

first

days

of

the

crawl

This

graph

contains

million

URLs

and

billion

links

between

them

We

then

compared

the

top

ranked

pages

between

the

two

data

sets

We

found

that

of

the

top

million

scoring

pages

were

downloaded

during

the

first

weeks

and

of

them

were

ranked

in

the

top

million

pages

in

the

day

data

set

That

is

it

was

clear

that

those

pages

were

important

even

before

the

crawl

had

finished

Figure

generalizes

these

statistics

for

each

value

of

N

we

plot

the

percentage

of

overlap

between

the

top

N

scoring

pages

in

the

day

crawl

versus

the

day

crawl

Although

the

top

few

pages

are

different

by

the

top

ranked

pages

there

is

an

overlap

The

overlap

continues

in

the

range

through

the

extent

of

the

entire

day

data

set

This

figure

suggests

that

breadth-first

search

crawling

is

fairly

immune

to

the

type

of

self-endorsement

described

above

although

the

size

of

the

graph

induced

by

the

full

crawl

is

about

larger

than

the

graph

induced

by

the

day

crawl

the

longer

crawl

replaced

only

about

of

the

hot

pages

discovered

during

the

first

days

irrespective

of

the

size

of

the

hot

set

Some

connectivity-based

metrics

such

as

Kleinberg

's

algorithm

-LSB-

-RSB-

consider

only

remote

links

that

is

links

between

pages

on

different

hosts

We

noticed

that

some

anomalies

in

Figure

were

due

to

a

lot

of

local

links

and

decided

to

experiment

with

a

variant

of

the

PageRank

algorithm

that

only

propagates

weights

along

remote

links

This

modification

of

PageRank

counts

only

links

from

different

hosts

as

proper

endorsements

of

a

page

links

from

the

same

host

are

viewed

as

improper

self-endorsement

and

therefore

not

counted

Figure

shows

our

results

the

average

PageRank

for

pages

downloaded

on

the

first

day

is

even

higher

than

when

all

links

are

considered

The

average

PageRank

for

the

first

day

is

while

it

's

on

the

second

day

and

on

the

top

N

pages

overlap

between

data

sets

Figure

The

percent

overlap

between

the

top

N

ranked

pages

in

the

first

vs

all

days

of

the

crawl

fourth

day

The

average

PageRank

then

declines

gradually

down

to

on

the

last

day

Notice

that

the

average

PageRank

on

the

first

day

of

crawling

is

higher

than

in

Figure

and

that

the

curve

falls

more

sharply

This

drop

indicates

that

our

crawling

strategy

is

not

biased

toward

self-endorsing

hosts

as

a

crawler

using

the

standard

version

of

PageRank

would

be

We

believe

that

this

lack

of

bias

is

due

in

part

to

our

crawler

's

politeness

policies

which

impose

a

rate

limit

on

its

accesses

to

any

particular

host

There

are

some

flaws

with

a

metric

based

only

on

remote

links

For

example

has

a

very

high

PageRank

score

However

it

only

has

local

outlinks

so

its

weight

gets

evenly

distributed

over

all

pages

in

the

graph

rather

than

just

to

the

other

pages

in

Yahoo!

to

which

it

points

Transitively

the

pages

on

other

hosts

to

which

Yahoo!

links

do

not

benefit

from

the

high

score

of

In

the

future

work

section

below

we

outline

some

ideas

for

remedying

this

problem

The

experiments

described

in

this

paper

demonstrate

that

a

crawler

that

downloads

pages

in

breadth-first

search

order

discovers

the

highest

quality

pages

during

the

early

stages

of

the

crawl

As

the

crawl

progresses

the

quality

of

the

downloaded

pages

deteriorates

We

speculate

that

breadth-first

search

is

a

good

crawling

strategy

because

the

most

important

pages

have

many

links

to

them

from

numerous

hosts

and

those

links

will

be

found

early

regardless

of

on

which

host

or

page

the

crawl

originates

Discovering

high-quality

pages

early

on

in

a

crawl

is

desirable

for

public

web

search

engines

such

as

AltaVista

or

Google

given

that

of

these

search

engines

is

able

to

crawl

and

index

more

than

a

fraction

of

the

web

Our

results

have

practical

implications

to

search

engine

companies

Although

breadth-first

search

crawling

seems

to

be

a

very

natural

crawling

strategy

not

all

of

the

crawlers

we

are

familiar

with

employ

it

For

example

the

Internet

Archive

crawler

described

in

-LSB-

-RSB-

does

not

perform

a

breadth-first

search

of

the

entire

web

instead

it

picks

hosts

at

a

time

and

crawls

these

hosts

in

parallel

Each

host

is

crawled

exhaustively

links

that

point

to

other

hosts

are

saved

to

seed

subsequent

crawls

of

those

hosts

This

crawling

strategy

has

no

bias

towards

high-quality

pages

if

the

hosts

to

be

crawled

are

picked

in

random

order

the

quality

of

downloaded

pages

will

be

uniform

throughout

the

crawl

Day

of

crawl

Average

PageRank

-LRB-

remote

links

only

-RRB-

Figure

Average

PageRank

when

only

remote

links

are

considered

Similarly

the

Scooter

web

crawler

used

until

recently

by

AltaVista

downloaded

pages

in

essentially

random

order

-LRB-

At

this

point

AltaVista

is

using

Mercator

-RRB-

This

approach

made

it

easier

to

provide

politeness

guarantees

essentially

it

spread

the

load

imposed

by

the

crawler

evenly

over

all

web

servers

but

as

a

result

the

quality

of

the

discovered

pages

is

uniform

over

the

life

of

the

crawl

We

can

not

make

any

statements

about

other

large-scale

web

crawlers

Most

search

engine

companies

treat

their

crawling

strategy

as

a

trade

secret

and

have

not

described

it

in

the

literature

Cho

et

al.

-LSB-

-RSB-

showed

that

using

a

connectivity-based

ordering

metric

for

downloads

such

as

PageRank

will

steer

the

crawler

towards

even

higher-quality

pages

than

using

breadth-first

search

However

computing

PageRank

values

for

several

hundred

million

or

more

pages

is

an

extremely

expensive

computation

It

took

us

over

a

day

to

compute

the

PageRanks

of

our

graph

of

million

pages

despite

the

fact

that

we

had

the

hardware

resources

to

hold

the

entire

graph

in

memory

Using

PageRank

to

steer

a

crawler

would

require

multiple

such

computations

over

larger

and

larger

graphs

in

order

to

take

newly

discovered

pages

into

account

and

is

essentially

infeasible

in

real

time

On

the

other

hand

crawling

in

breadth-first

search

order

provides

a

fairly

good

bias

towards

high

quality

pages

without

the

computational

cost

We

believe

that

crawling

in

breadth-first

search

order

provides

the

better

tradeoff

There

are

two

directions

in

which

we

would

like

to

extend

this

work

One

direction

is

to

try

a

variant

of

PageRank

which

weighs

links

to

pages

on

remote

hosts

differently

than

links

to

other

pages

on

the

same

host

From

the

experiment

that

generated

Figure

above

we

learned

that

remote

links

should

count

more

than

local

links

but

that

weights

should

be

propagated

along

local

links

as

well

-LRB-

e.g.

to

distribute

the

weight

of

to

the

pages

that

Yahoo!

recommends

-RRB-

We

suspect

that

some

search

engines

already

use

different

weights

for

links

but

there

has

been

no

formal

study

of

how

to

divide

the

weights

among

the

links

or

even

whether

the

division

should

be

static

-LRB-

e.g.

remote

links

get

of

the

total

weight

-RRB-

or

proportional

to

the

number

of

total

links

-LRB-

e.g.

each

remote

link

gets

four

times

the

weight

of

each

local

link

-RRB-

The

other

direction

is

to

try

different

connectivity-based

metrics

While

PageRank

is

the

only

connectivity

measure

we

know

aimed

at

ranking

all

of

the

pages

on

the

world

wide

web

Kleinberg

's

algorithm

-LSB-

-RSB-

is

another

well-known

connectivity

analysis

algorithm

targeted

towards

computing

quality

scores

for

pages

The

algorithm

computes

two

scores

for

each

document

a

hub

score

and

an

authority

score

Pages

with

high

authority

scores

are

expected

to

have

high-quality

content

the

authority

scores

are

similar

in

intent

to

PageRanks

Kleinberg

's

algorithm

is

designed

to

rank

the

results

of

a

query

to

a

search

engine

and

only

considers

a

small

set

of

pages

when

it

computes

authority

scores

However

we

believe

that

we

can

extend

the

algorithm

to

consider

the

entire

graph

of

the

web

-LSB-

-RSB-

K.

Bharat

A.

Broder

M.

Henzinger

P.

Kumar

and

S.

Venkatasubramanian

The

connectivity

server

Fast

access

to

linkage

information

on

the

web

In

Proceedings

of

the

International

World

Wide

Web

Conference

pages

Brisbane

Australia

April

Elsevier

Science

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

anatomy

of

a

large-scale

hypertextual

web

search

engine

In

Proceedings

of

the

International

World

Wide

Web

Conference

pages

Brisbane

Australia

April

Elsevier

Science

-LSB-

-RSB-

M.

Burner

Crawling

towards

eternity

Building

an

archive

of

the

world

wide

web

Web

Techniques

Magazine

-LRB-

-RRB-

May

-LSB-

-RSB-

J.

Cho

H.

Garcia-Molina

and

L.

Page

Efficient

crawling

through

URL

ordering

In

Proceedings

of

the

International

World

Wide

Web

Conference

pages

Brisbane

Australia

April

Elsevier

Science

-LSB-

-RSB-

Google

Inc.

Press

release

Google

launches

world

's

largest

search

engine

June

Available

at

-LSB-

-RSB-

M.

Henzinger

A.

Heydon

M.

Mitzenmacher

and

M.

Najork

On

near-uniform

URL

sampling

In

Proceedings

of

the

International

World

Wide

Web

Conference

pages

Amsterdam

Netherlands

May

Elsevier

Science

-LSB-

-RSB-

A.

Heydon

and

M.

Najork

Mercator

A

scalable

extensible

web

crawler

World

Wide

Web

-LRB-

-RRB-

Dec.

-LSB-

-RSB-

J.

Kleinberg

Authoritative

sources

in

a

hyperlinked

environment

In

Proceedings

of

the

ACM-SIAM

Symposium

on

Discrete

Algorithms

pages

San

Francisco

CA

Jan.

-LSB-

-RSB-

P.

Lyman

H.

Varian

J.

Dunn

A.

Strygin

and

K.

Swearingen

How

much

information

School

of

Information

Management

and

Systems

Univ.

of

California

at

Berkeley

Available

at

-LSB-

-RSB-

Mercator

Home

Page

-LSB-

-RSB-

J.

L.

Wiener

R.

Wickremesinghe

M.

Burrows

K.

Randall

and

R.

Stata

Better

link

compression

Manuscript

in

progress

Compaq

Systems

Research

Center

VITAE

Marc

Najork

is

a

senior

member

of

the

research

staff

at

Compaq

Computer

Corporation

's

Systems

Research

Center

His

current

research

focuses

on

high-performance

web

crawling

and

web

characterization

He

was

a

principal

contributor

to

Mercator

the

web

crawler

used

by

AltaVista

In

the

past

he

has

worked

on

tools

for

web

surfing

animation

information

visualization

algorithm

animation

and

visual

programming

languages

He

received

a

Ph.D.

in

Computer

Science

from

the

University

of

Illinois

at

Urbana-Champaign

in

Janet

L.

Wiener

is

a

member

of

the

research

staff

at

Compaq

Computer

Corporation

's

Systems

Research

Center

She

currently

focuses

on

developing

algorithms

to

characterize

the

web

and

tools

-LRB-

such

as

the

Connectivity

Server

-RRB-

to

support

those

algorithms

Prior

to

joining

Compaq

in

she

was

a

research

scientist

at

Stanford

University

working

on

data

warehousing

heterogeneous

data

integration

and

semi-structured

data

She

received

a

Ph.D.

from

the

University

of

Wisconsin-Madison

in

and

a

B.A.

from

Williams

College

in

O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

crawling
high quality pages
ordering metrics
pagerank
page quality metric

breadth first search
crawl order
breadth-first search
connectivity-based metrics


A Flexible 3D Slicer for Voxelization Using Graphics Hardware In this paper we present a simple but general 3D slicer for voxelizing polygonal model . Instead of voxelizing a model by projecting and rasterizing triangles with clipping planes , the distance field is used for more accurate and stable voxelization . Distance transform is used with triangles on voxels of each slice . A voxel is marked with opacity only when the shortest distance between it and triangles is judged as intersection . With advanced programmable graphics hardware assistance , surface and solid voxelization are feasible and more efficient than on a CPU . CR Categories : I. 3.1 -LSB- Computer Graphics -RSB- : Hardware Architecture -- Graphics processors ; I. 3.5 -LSB- Computer Graphics -RSB- : Computational Geometry and Object Modeling -- Curve , surface , solid , and object representations ; Object representation is a broad topic in research . In computer graphics , polygons play a dominant role in 3D graphics because they approximate arbitrary surfaces by meshes . In games and animations , surface representation is the main technique used in rendering and visualization . However , volumetric representation , an alternative method to traditional geometric representation , is well known since the 1980s . It provides a simple and uniform description to measure and model a volumetric objects and establish the research field of volumetric graphics . Voxelization is a process of constructing a volumetric representation of an object . Voxelizing a polygonal object is not only a shift of the representation , it gives an opportunity to manipulate mesh object with volumetric operations such as morphological operation and solid modeling . Many applications , for example , CSG modeling , virtual medicine , haptic rendering , visualization of geometric model , collision detection , 3D spatial analysis , and model fixing , work on volumetric representation or use it as the inter-medium . In this paper , we calculate the accurate distance field on GPU to compute coverage of each voxel in voxelization for polygonal models . Our method works for arbitrary triangulated models without any preprocessing for models , except organizing meshes slab by slab in order to prune the unnecessary computation while voxelizing complex models . By using the power of GPU , Hausdorff distance is guaranteed between each sampled voxel and the polygonal model . Surface voxelization with distance field on a GPU works well and runs faster than on a PC with a CPU . Our method is a reliable and accurate solution for the polygonal model under a given distribution of sampling voxels and a specific opacity criterion . Besides , error tolerance in voxelization is easy to manipulate by adjusting the threshold of opacity criterion for voxelization , which also dominates the smoothness of features and the thickness of surface voxelization . e-mail : wktai@mail.ndhu.edu.tw The rest of paper is organized as follows . Some related works are surveyed in the section 2 . In section 3 , we present the computation of Hausdorff distance and our framework . The experimental results are illustrated in section 4 . Finally , we conclude the proposed approach and point out some future works . Volume construction approaches are often referred to as scan-conversion or voxelization methods . Researchers mainly focused on modeling aspects such as robustness and accuracy . Wang and Kaufman -LSB- Wang and Kaufman 1993 -RSB- used a method that samples and filters the voxels in 3D space to produce alias-free 3D volume models . They used filters to produce final density from the support of the region that polygons lie inside . Schroeder and Lorensen -LSB- Schroeder et al. 1994 -RSB- created a volumetric model by finding clos-est polygon from distance map and classify the opacity of voxels . Huang et al. -LSB- Huang et al. 1998 -RSB- described separability and min-imality as two desirable features of a discrete surface representation and extended 2D scan line algorithm to perform voxelization . Dachille and Kaufman -LSB- Dachille IX and Kaufman 2000 -RSB- presented an incremental method for voxelizing a triangle with pre-filtering to generate multivalued voxelization . Widjaya et al. -LSB- Widjaya et al. 2003 -RSB- presented the voxelization in common sampling lattices as general 2D lattices including hexagonal lattices and 3D body-center cubic lattices . Ju -LSB- Ju 2004 -RSB- constructed an octree grid for recording intersected edges with the model and expanded nodes to scan-convert a polygon on an octree , and then generate signs from the boundary edges and faces of the octree . In recent years , attention on performance of voxelization raises . More and more studies try to explore the benefits of graphics hardware for more efficient rendering . Chen and Feng -LSB- Chen and Fang 1999 -RSB- presented a slicing-based voxelization algorithm to generate slices of the underlying model in the frame buffer by setting appropriate clipping planes and extracting each slice of the model , which is extended and published in later -LSB- Chen and Fang 2000 -RSB- . Karabassi and Theoharis -LSB- Karabassi and Theoharis 1999 -RSB- projected an object to six faces of its bounding box through standard graphics system for the outermost parts and read back the information from depth buffer . However it works well only on convex objects . Dong et al. -LSB- Dong et al. 2004 -RSB- proposed a real-time voxelization method using GPU acceleration to rasterize and texelize an object into three directional textures and then synthesize textures back to the final volume . 285 In this section , we first discuss the computation of Hausdorff distance between a given triangle and a point . Then we explain how we use GPU to compute the distance field of triangles and modify the rendering pipeline . 3.1 Distance Field Computation For a given 3D point P -LRB- x , y , z -RRB- and a triangle T -LRB- V 0 , V 1 , V 2 -RRB- , Hausdorff distance is the shortest distance between the point P and any point v on the triangle . A point on triangle T can be parametrically defined by two linearly independent vectors with two weights -LRB- s , t -RRB- by T -LRB- s , t -RRB- = B + se 0 + te 1 , -LRB- 1 -RRB- where -LRB- s , t -RRB- D = -LCB- -LRB- s , t -RRB- : s -LSB- 0 , 1 -RSB- , t -LSB- 0 , 1 -RSB- , s + t 1 -RCB- , and B = V 0 , e 0 = V 1 - V 0 and e 1 = V 2 - V 0 . For any point on triangle T , the distance from T to P is T -LRB- s , t -RRB- - P , -LRB- 2 -RRB- or we can use the squared-distance function instead Q -LRB- s , t -RRB- = T -LRB- s , t -RRB- - P 2 , -LRB- 3 -RRB- where a point p = -LRB- s , t -RRB- exists which makes Q -LRB- s , t -RRB- minimum . Therefore , the computation of distance can be reduced into a min-imization problem . For an efficient computation , we can expand Q -LRB- s , t -RRB- as Q -LRB- s , t -RRB- = as 2 + 2bst + ct 2 + 2ds + 2et + f , -LRB- 4 -RRB- where a = e 0 e 0 b = e 0 e 1 c = e 1 e 1 d = e 0 -LRB- B - P -RRB- e = e 1 -LRB- B - P -RRB- f = -LRB- B - P -RRB- -LRB- B - P -RRB- -LRB- 5 -RRB- From analyzing the gradient of Q -LRB- s , t -RRB- , the minimum s and t happens only when Q is zero , where s = be - cd ac - b 2 t = bd - ae ac - b 2 -LRB- 6 -RRB- If -LRB- s , t -RRB- D , the minimum distance is the distance between p and P ; otherwise , according to the sign of s and t , there are six possible regions that the shortest distance point p may lie on triangle T , as shown in Figure 1 . Efficient solutions are well addressed on the book -LSB- Schneider and Eberly 2003 -RSB- by CPU computation with simple calculation and logic classification . However , in GPU , there is no efficient dynamic flow control to determine the shortest point on a triangle . Therefore , instead of directly computing the point of shortest distance on a triangle , we compute the distance from the 3D point to four possible points which may be inside the triangle or on the three boundaries and then the minimum scalar is the shortest distance . These four points are -LRB- s 0 , t 0 -RRB- = -LRB- be - cd ac - b 2 , bd - ae ac - b 2 -RRB- -LRB- s 1 , t 1 -RRB- = -LRB- 0 , e c -RRB- -LRB- s 3 , t 3 -RRB- = -LRB- c + e-b-d a -2 b + c , a + d-b-e a -2 b + c -RRB- -LRB- s 2 , t 2 -RRB- = -LRB- d a , 0 -RRB- , -LRB- 7 -RRB- Figure 1 : Six regions in s , t coordinate . Space is partitioned by range of parameters s and t for efficient shortest position classification and calculation . Figure 2 : Rendering pipeline for generating a distance field of a triangle . A quad is rendered instead of a triangle . Five channels of each vertex of a quad -LRB- position , normal , and 4 texture coordinates -RRB- is filled with position of quad , position of voxel , and information of a triangle : v 0 , e 0 , e 1 , and normal N respectively . where position -LRB- s 0 , t 0 -RRB- assumes point p is inside the triangle , positions -LRB- s 1 , t 1 -RRB- , -LRB- s 2 , t 2 -RRB- and -LRB- s 3 , t 3 -RRB- assume point p is on boundaries of s = 0 , t = 0 , and s + t = 1 . All calculated points are truncated in the range of -LSB- 0 , 1 -RSB- so that three end vertices of the triangle T are also in consideration and it guarantees these points are on the triangle for distance computation . Therefore , the minimum distance is the shortest distance from the point P to the triangle T . 3.2 Geometry Rendering for Voxelization Voxelization by projection and rasterization faces the difficulty of non-uniform sampling of polygons because polygons with arbitrary orientations are not parallel to projection plane for maximum projection area . Even classifying polygons and projecting them to individual best-fit plane , there still have no guarantee on valid rasterization . However , distance field is omni-directional , i.e. , insensitive to the projection plane , and has no assumption on input geometry and therefore no extra preprocessing is required . Our approach is a slice-based approach for distance field generation and voxelization . Figure 2 shows the rendering process for generating the distance field for a triangle . For each triangle T i = -LCB- T i -LRB- s , t -RRB- | v 0 + se 0 + te 1 , s 0 , t 0 , s + t 1 -RCB- , a full-filled quad Q i = -LCB- q i 0 , q i 1 , q i 2 , q i 3 -RCB- is rendered and rasterized to generate the distance field from voxels on a slice to the triangle . Triangle data and voxel positions are associated with the rendering quads . Voxel positions are stored in the channel of vertex normal , the triangle data -LRB- base vertex B , vectors e 0 and te 1 , and the normal N -RRB- are separately stored in channels of texture coordinates and transmitted to GPU . Voxel positions are linearly interpolated in rasterization of rendering pipeline and pairs of triangle data and voxel positions are sent to pixel processors for Hausdorff distance computation . After distance computation , the shortest distance between a triangle and a 286 voxel is stored in the pixel depth and the pixel color is assigned for identification depending on applications . For example , binary surface voxelization uses color information to identify whether a voxel intersects a geometry such as 0 for empty and 1 for opacity ; distance visualization uses color information to display the distance from geometries , etc. . The distance field of polygonal objects is constructed incrementally by rendering quads of triangles . Each pixel of depth buffer keeps the shortest distance from its voxel to triangles rendered . Depth buffer updates when a triangle is rendered . Unless distance is recal-culated on different slice or rendered objects deform , quads which have been rendered have no need to be re-rendered again even new geometry are added in the scene . Depth buffer of the viewport is initialized as infinitude . The rendering pseudo code is abstracted as follows : for each triangle t on slice i -LCB- Create a quad Q for the triangle t for k = 0 to 3 -LCB- / / assign a full-filled quad / / q is end vertices of quad Q.q -LSB- k -RSB- . position = ScreenBoundary.q -LSB- k -RSB- ; / / assign voxel position , and triangle data Q.q -LSB- k -RSB- . normal = Slice -LSB- i -RSB- . q -LSB- k -RSB- ; Q.q -LSB- k -RSB- . tex0 = t.B ; Q.q -LSB- k -RSB- . tex1 = t.e0 ; Q.q -LSB- k -RSB- . tex2 = t.e1 ; -RCB- RenderQuad -LRB- Q -RRB- ; -RCB- 3.3 Surface Voxelization We use local distance field to reduce work load of GPU because the distance field far away from a triangle is meaningless for surface voxelization . For each triangle , we extend its parallel projected bounding rectangle by a small scalar for effective rasterization especially for triangles perpendicular to the projection plane . Due to coherence and precision in interpolating voxel positions , triangles are rendered with extended bounding rectangles . While a pixel is rasterized by a quad , Hausdorff distance is calculated according to the interpolated voxels , i.e. , centers of voxels , and the triangle data . Only if the distance is less than the given threshold , e.g. , distance from a uniform voxel center to its corner , the pixel is marked as opacity . Using local distance field could guarantee a small region of Hausdorff distance but greatly improve the performance of surface voxelization . For more efficient voxelization process on GPU , triangles can be culled early by space partitioning . We construct an active triangle list for each slice . Currently we define slabs along Z-axis . According to partitioning planes , triangles are filtered and rearranged slab by slab . Many triangles can be pruned while rendering a slice . It is significantly helpful while voxelizing very complex models . Because distance field is insensitive to projecting directions of triangles , selection of partitioning plane has no influence on effective-ness of voxelization . We implement our fragment program using HLSL on a Pentium 4 3.0 MHz PC with 1G RAM and a nVidia Geforce FX5800 graphics card running on Windows XP with DirectX 9.0 c . We use Vertex Shader 1.1 and Pixel Shader 2.0 to implement fragment program in scattering pairs of voxel positions and triangle data and in distance calculation and visualization . Table 1 shows the performance Figure 4 : Rendering from the results of voxelization -LRB- 512 3 -RRB- : dragon of 870K faces in 512 3 voxels . of surface voxelization on different models and in different voxel resolutions . Figure 3 and Figure 4 demonstrate quality of voxelization results . In the experiment , opacity threshold is set to the distance from voxel center to its corner . That means if the shortest distance between a voxel and a triangle is less than the threshold , the voxel will be marked as opacity . Note that voxels are normal-ized to cubes in rendering so the scale of output may differ from the original polygonal model . In Table 1 , we list average time on surface voxelization per slice , per voxel , and per triangle . In the same resolution , voxelization time is proportional to the complexity of polygonal model . For each voxel , process time is always less than 0.1 ms. Even when voxel resolution increase , GPU still could handle voxelization for complex object in stable throughput which may be increased much more for CPU . Due to speed up by using local distance field and culling for unre-lated geometry , voxelization by distance field can be displayed slice by slice interactively under 128 3 voxel resolution . When voxel resolution is low , voxelization time highly depends on complexity of model . However , when the voxel resolution increases higher , even for a low complexity model , it still need more time to voxelize a model than in lower voxel resolution . On average , resolution of 256 3 could provide a benefit of reliable voxelization both in quality and time cost . Rendering cost is stable for a triangle even when resolution of volume increases while it is linear on a CPU . Voxelization with proposed method is still slower than methods using traditional projection and rasterization by graphic hardware . However , our method is stable , correct and flexible because the opacity of each voxel is determined by thresholding the distance field of objects . In this paper , we propose a GPU-based 3D slicer approach for voxelizing polygonal models . We calculate minimum distance between pairs of sampled voxels and triangles of arbitrary models with guarantee of Hausdorff distance . With programmable hardware ver-tex/pixel processors , efficient surface voxelization , solid voxelization , and visualization of the distance field all are feasible on the proposed 3D slicer . However , in current implementation , performance of pixel shader is the bottleneck in overall processing speed . Area of rasterization also has a significant influence on the loading of pixel shader . 287 Model Faces Res . Time -LRB- s -RRB- Time Slices Time Voxels Time Tri . Res . Time -LRB- s -RRB- Time Slices Time Voxels Time Tri . Beethoven 5027 128 13.94 0.11 6.65 2.77 256 85.86 0.34 5.12 17.08 Teapot 6320 128 14.31 0.11 6.82 2.26 256 87.62 0.34 5.22 13.86 Cup 7494 128 15.24 0.12 7.27 2.03 256 93.65 0.37 5.58 12.50 Bunny 10000 128 16.24 0.13 7.75 1.62 256 93.98 0.37 5.60 9.40 Bunny 69451 128 43.21 0.34 20.60 0.62 256 231.85 0.91 13.82 3.34 Dragon 871414 128 84.21 0.66 40.15 0.10 256 325.87 1.27 19.42 0.37 Buddha 1087716 128 170.44 1.33 81.27 0.16 256 347.65 1.36 20.72 0.32 Dragon 871414 512 1748.11 3.41 13.02 2.01 * The time unit in Time/Voxels is s Buddha 1087716 512 1825.47 3.57 13.60 1.68 * The time unit in Time/Tri . is ms Table 1 : Surface voxelization on different models and in different voxel resolutions . -LRB- a -RRB- -LRB- b -RRB- -LRB- c -RRB- -LRB- d -RRB- -LRB- e -RRB- -LRB- f -RRB- -LRB- g -RRB- -LRB- h -RRB- Figure 3 : Rendering from the results of voxelization -LRB- 256 3 -RRB- : -LRB- a -RRB- Beethoven in 256 3 voxels , -LRB- b -RRB- Teapot in 256 3 voxels , -LRB- c -RRB- Cup in 256 3 voxels , -LRB- d -RRB- Bunny of 10000 faces in 256 3 voxels , -LRB- e -RRB- dragon of 10000 faces in 256 3 voxels , -LRB- f -RRB- Bunny of 69451 faces in 256 3 voxels , -LRB- g -RRB- dragon of 870K faces in 256 3 voxels , and -LRB- h -RRB- Buddha of 1M faces in 256 3 voxels . Therefore , in the near feature , searching a better computational methodology for GPU is one direction to improve performance of distance field computation . In addition , a sophisticated culling for error-free distance computation will be a technique in demand . To improve the quality of voxelization , adaptive dense voxelization and a mechanism for quality measurement and guide on GPU is another interesting topic . C HEN , H. , AND F ANG , S. 1999 . Fast voxelization of 3D synthetic objects . ACM Journal of Graphics Tools 3 , 4 , 33 45 . C HEN , H. , AND F ANG , S. 2000 . Hardware accelerated voxelization . Computers and Graphics 24 , 3 , 433 442 . D ACHILLE IX , F. , AND K AUFMAN , A. E. 2000 . Incremental triangle voxelization . In Graphics Interface , 205 212 . D ONG , Z. , C HEN , W. , B AO , H. , Z HANG , H. , AND P ENG , Q. 2004 . Real-time voxelization for complex polygonal models . In Proceedings of Pacific Graphics '04 , 43 50 . H UANG , J. , Y AGEL , R. , F ILIPPOV , V. , AND K URZION , Y. 1998 . An accurate method for voxelizing polygon meshes . In Proceedings of IEEE symposium on Volume visualization , 119 126 . J U , T. 2004 . Robust repair of polygonal models . ACM Transactions on Graphics 23 , 3 , 888 895 . K ARABASSI , G. P. E.-A. , AND T HEOHARIS , T. 1999 . A fast depth-buffer-based voxelization algorithm . ACM Journal of Graphics Tools 4 , 4 , 5 10 . S CHNEIDER , P. , AND E BERLY , D. H. 2003 . Geometry Tools for Computer Graphics . Morgan Kaufmann . S CHROEDER , W. J. , L ORENSEN , W. E. , AND L INTHICUM , S. 1994 . Implicit modeling of swept surfaces and volumes . In Proceedings of IEEE Visualization , 40 45 . W ANG , S. W. , AND K AUFMAN , A. E. 1993 . Volume sampled voxelization of geometric primitives . In Proceedings of IEEE Visualization , 78 84 . W IDJAYA , H. , M UELLER , T. , AND E NTEZARI . , A. 2003 . Voxelization in common sampling lattices . In Proceedings of Pacific Graphics '03 , 497 501 . 288

O O B I O B O B I O O O O O O O O O B I O O B I O O O O O O O O O O B O O O O O B I O O O O O O O B O B I O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O B I O O B O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O B I O O B O B I O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O B O O O O O O O O O B O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O B I O B O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O B I O O O O B O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I I O O O O O B O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I I O O O O O O B I O O O O O O O B O O O O B O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O B O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O B O O B O O O O O B I O O O O O O B I O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O

hausdorff distance
gpu
triangles
polygonal model
object representation
geometric representation
modeling
3d slicer
graphics hardware
voxelization
rendering cost
polygonal objects
local distance field
rendering
surface representation
infinitude
computer graphics
volume construction
distance transform
computational geometry
polygonal object
pixel shader
volumetric representation
distance field
rasterization
opacity

hausforff distance
slice-based approach
gpu computation
3d modelling
surface voxelization
adaptive dense voxelization
rendering pipeline
gpu-based 3d slicer approach