Parallel

Crawlers

In

this

paper

we

study

how

we

can

design

an

effective

parallel

crawler

As

the

size

of

the

Web

grows

it

becomes

imperative

to

parallelize

a

crawling

process

in

order

to

finish

downloading

pages

in

a

reasonable

amount

of

time

We

first

propose

multiple

architectures

for

a

parallel

crawler

and

identify

fundamental

issues

related

to

parallel

crawling

Based

on

this

understanding

we

then

propose

metrics

to

evaluate

a

parallel

crawler

and

compare

the

proposed

architectures

using

million

pages

collected

from

the

Web

Our

results

clarify

the

relative

merits

of

each

architecture

and

provide

a

good

guideline

on

when

to

adopt

which

architecture

H.

-LSB-

Hypertext/Hypermedia

-RSB-

Architectures

H.

-LSB-

Systems

and

Software

-RSB-

Distributed

Systems

Management

Performance

Design

A

crawler

is

a

program

that

downloads

and

stores

Web

pages

often

for

a

Web

search

engine

Roughly

a

crawler

starts

off

by

placing

an

initial

set

of

URLs

S

in

a

queue

where

all

URLs

to

be

retrieved

are

kept

and

prioritized

From

this

queue

the

crawler

gets

a

URL

-LRB-

in

some

order

-RRB-

downloads

the

page

extracts

any

URLs

in

the

downloaded

page

and

puts

the

new

URLs

in

the

queue

This

process

is

repeated

until

the

crawler

decides

to

stop

Collected

pages

are

later

used

for

other

applications

such

as

a

Web

search

engine

or

a

Web

cache

As

the

size

of

the

Web

grows

it

becomes

more

difficult

to

retrieve

the

whole

or

a

significant

portion

of

the

Web

using

a

single

process

Therefore

many

search

engines

often

run

multiple

processes

in

parallel

to

perform

the

above

task

so

that

download

rate

is

maximized

We

refer

to

this

type

of

crawler

as

a

parallel

crawler

In

this

paper

we

study

how

we

should

design

a

parallel

crawler

so

that

we

can

maximize

its

performance

-LRB-

e.g.

Copyright

is

held

by

the

author/owner

-LRB-

s

-RRB-

May

Honolulu

Hawaii

USA

ACM

download

rate

-RRB-

while

minimizing

the

overhead

from

parallelization

We

believe

many

existing

search

engines

already

use

some

sort

of

parallelization

but

there

has

been

little

scientific

research

conducted

on

this

topic

Thus

little

has

been

known

on

the

tradeoffs

among

various

design

choices

for

a

parallel

crawler

In

particular

we

believe

the

following

issues

make

the

study

of

a

parallel

crawler

challenging

and

interesting

Overlap

When

multiple

processes

run

in

parallel

to

download

pages

it

is

possible

that

different

processes

download

the

same

page

multiple

times

One

process

may

not

be

aware

that

another

process

has

already

downloaded

the

page

Clearly

such

multiple

downloads

should

be

minimized

to

save

network

bandwidth

and

increase

the

crawler

's

effectiveness

Then

how

can

we

coordinate

the

processes

to

prevent

overlap

Quality

Often

a

crawler

wants

to

download

important

pages

first

in

order

to

maximize

the

quality

of

the

downloaded

collection

However

in

a

parallel

crawler

each

process

may

not

be

aware

of

the

whole

image

of

the

Web

that

they

have

collectively

downloaded

so

far

For

this

reason

each

process

may

make

a

crawling

decision

solely

based

on

its

own

image

of

the

Web

-LRB-

that

itself

has

downloaded

-RRB-

and

thus

make

a

poor

crawling

decision

Then

how

can

we

make

sure

that

the

quality

of

the

downloaded

pages

is

as

good

for

a

parallel

crawler

as

for

a

centralized

one

Communication

bandwidth

In

order

to

prevent

overlap

or

to

improve

the

quality

of

the

downloaded

pages

crawling

processes

need

to

periodically

communicate

to

coordinate

with

each

other

However

this

communication

may

grow

significantly

as

the

number

of

crawling

processes

increases

Exactly

what

do

they

need

to

communicate

and

how

significant

would

this

overhead

be

Can

we

minimize

this

communication

overhead

while

maintaining

the

effectiveness

of

the

crawler

While

challenging

to

implement

we

believe

that

a

parallel

crawler

has

many

important

advantages

compared

to

a

single-process

crawler

Scalability

Due

to

enormous

size

of

the

Web

it

is

often

imperative

to

run

a

parallel

crawler

A

single-process

crawler

simply

can

not

achieve

the

required

download

rate

in

certain

cases

Network-load

dispersion

Multiple

crawling

processes

of

a

parallel

crawler

may

run

at

geographically

distant

locations

each

downloading

geographically-adjacent

pages

For

example

a

process

in

Germany

may

download

all

European

pages

while

another

one

in

Japan

crawls

all

Asian

pages

In

this

way

we

can

disperse

the

network

load

to

multiple

regions

In

particular

this

dispersion

might

be

necessary

when

a

single

network

can

not

handle

the

heavy

load

from

a

large-scale

crawl

Network-load

reduction

In

addition

to

the

dispersing

load

a

parallel

crawler

may

actually

reduce

the

network

load

For

example

assume

that

a

crawler

in

North

America

retrieves

a

page

from

Europe

To

be

downloaded

by

the

crawler

the

page

first

has

to

go

through

the

network

in

Europe

then

the

Europe-to-North

America

inter-continental

network

and

finally

the

network

in

North

America

Instead

if

a

crawling

process

in

Europe

collects

all

European

pages

and

if

another

process

in

North

America

crawls

all

North

American

pages

the

overall

network

load

will

be

reduced

because

pages

go

through

only

local

networks

Note

that

the

downloaded

pages

may

need

to

be

transferred

later

to

a

central

location

so

that

a

central

index

can

be

built

However

even

in

that

case

we

believe

that

the

transfer

can

be

significantly

smaller

than

the

original

page

download

traffic

by

using

some

of

the

following

methods

Compression

Once

the

pages

are

collected

and

stored

it

is

easy

to

compress

the

data

before

sending

them

to

a

central

location

Difference

Instead

of

sending

the

entire

image

with

all

downloaded

pages

we

may

first

take

difference

between

previous

image

and

the

current

one

and

send

only

this

difference

Since

many

pages

are

static

and

do

not

change

very

often

this

scheme

can

significantly

reduce

the

network

traffic

Summarization

In

certain

cases

we

may

need

only

a

central

index

not

the

original

pages

themselves

In

this

case

we

may

extract

the

necessary

information

for

the

index

construction

-LRB-

e.g.

postings

list

-RRB-

and

transfer

this

data

only

To

build

an

effective

web

crawler

we

clearly

need

to

address

many

more

challenges

than

just

parallelization

For

example

a

crawler

needs

to

figure

out

how

often

a

page

changes

and

how

often

it

would

revisit

the

page

in

order

to

maintain

the

page

up

to

date

-LSB-

-RSB-

Also

it

has

to

make

sure

that

a

particular

Web

site

is

not

flooded

with

its

HTTP

requests

during

a

crawl

-LSB-

-RSB-

In

addition

it

has

to

carefully

select

what

page

to

download

and

store

in

its

limited

storage

space

in

order

to

make

the

best

use

of

its

stored

collection

of

pages

-LSB-

-RSB-

While

all

of

these

issues

are

important

we

focus

on

the

crawler

parallelization

in

this

paper

because

this

problem

has

been

paid

significantly

less

attention

than

the

others

In

summary

we

believe

a

parallel

crawler

has

many

advantages

and

poses

interesting

challenges

In

particular

we

believe

our

paper

makes

the

following

contributions

We

identify

major

issues

and

problems

related

to

a

parallel

crawler

and

discuss

how

we

can

solve

these

problems

We

present

multiple

techniques

for

a

parallel

crawler

and

discuss

their

advantages

and

disadvantages

As

far

as

we

know

most

of

these

techniques

have

not

been

described

in

open

literature

-LRB-

Very

little

is

known

about

the

internals

of

crawlers

as

they

are

closely

guarded

secrets

-RRB-

Using

a

large

dataset

-LRB-

web

pages

-RRB-

collected

from

the

Web

we

experimentally

compare

the

design

choices

and

study

their

tradeoffs

quantitatively

We

propose

various

optimization

techniques

that

can

minimize

the

coordination

effort

between

crawling

processes

so

that

they

can

operate

more

independently

while

maximizing

their

effectiveness

Related

work

Web

crawlers

have

been

studied

since

the

advent

of

the

Web

-LSB-

-RSB-

These

studies

can

be

roughly

categorized

into

one

of

the

following

topics

General

architecture

-LSB-

-RSB-

The

work

in

this

category

describes

the

general

architecture

of

a

Web

crawler

and

studies

how

a

crawler

works

For

example

Reference

-LSB-

-RSB-

describes

the

architecture

of

the

Compaq

SRC

crawler

and

its

major

design

goals

Some

of

these

studies

briefly

describe

how

the

crawling

task

is

parallelized

For

instance

Reference

-LSB-

-RSB-

describes

a

crawler

that

distributes

individual

URLs

to

multiple

machines

which

download

Web

pages

in

parallel

The

downloaded

pages

are

then

sent

to

a

central

machine

on

which

links

are

extracted

and

sent

back

to

the

crawling

machines

However

these

studies

do

not

carefully

compare

various

issues

related

to

a

parallel

crawler

and

how

design

choices

affect

performance

In

this

paper

we

first

identify

multiple

techniques

for

a

parallel

crawler

and

compare

their

relative

merits

using

real

Web

data

Page

selection

-LSB-

-RSB-

Since

many

crawlers

can

download

only

a

small

subset

of

the

Web

crawlers

need

to

carefully

decide

what

page

to

download

By

retrieving

important

or

relevant

pages

early

a

crawler

may

improve

the

quality

of

the

downloaded

pages

The

studies

in

this

category

explore

how

a

crawler

can

discover

and

identify

important

pages

early

and

propose

various

algorithms

to

achieve

this

goal

In

our

paper

we

study

how

parallelization

affects

some

of

these

techniques

and

explain

how

we

can

fix

the

problems

introduced

by

parallelization

Page

update

-LSB-

-RSB-

Web

crawlers

need

to

update

the

downloaded

pages

periodically

in

order

to

maintain

the

pages

up

to

date

The

studies

in

this

category

discuss

various

page

revisit

policies

to

maximize

the

freshness

of

the

downloaded

pages

For

example

Reference

-LSB-

-RSB-

studies

how

a

crawler

should

adjust

revisit

frequencies

for

pages

when

the

pages

change

at

different

rates

We

believe

these

studies

are

orthogonal

to

what

we

discuss

in

this

paper

There

also

exists

a

significant

body

of

literature

studying

the

general

problem

of

parallel

and

distributed

computing

-LSB-

-RSB-

Some

of

these

studies

focus

on

the

design

of

efficient

parallel

algorithms

For

example

References

-LSB-

-RSB-

present

various

architectures

for

parallel

computing

propose

algorithms

that

solve

various

problems

-LRB-

e.g.

finding

maximum

cliques

-RRB-

under

the

architecture

and

study

the

complexity

of

the

proposed

algorithms

While

the

general

principles

described

are

being

used

in

our

work

of

the

existing

solutions

can

be

directly

applied

to

the

crawling

problem

Another

body

of

literature

designs

and

implements

distributed

operating

systems

where

a

process

can

use

distributed

resources

transparently

-LRB-

e.g.

distributed

memory

distributed

file

systems

-RRB-

-LSB-

-RSB-

Clearly

such

OS-level

support

makes

it

easy

to

build

a

general

distributed

application

but

we

believe

that

we

can

not

simply

run

a

centralized

crawler

on

a

distributed

OS

to

achieve

parallelism

A

web

crawler

contacts

millions

of

web

sites

in

a

short

period

of

time

and

consumes

extremely

large

network

storage

and

memory

resources

Since

these

loads

push

the

limit

of

existing

hardwares

the

task

should

be

carefully

partitioned

among

processes

and

they

should

be

carefully

coordinated

Therefore

a

general-purpose

distributed

operating

system

that

does

not

understand

the

semantics

of

web

crawling

will

lead

to

unacceptably

poor

performance

In

Figure

we

illustrate

the

general

architecture

of

a

parallel

crawler

A

parallel

crawler

consists

of

multiple

crawling

processes

which

we

refer

to

as

C-proc

's

Each

C-proc

performs

the

basic

tasks

that

a

single-process

crawler

conducts

It

downloads

pages

from

the

Web

stores

the

pages

locally

extracts

URLs

from

the

downloaded

pages

and

follows

links

Depending

on

how

the

C-proc

's

split

the

download

task

some

of

the

extracted

links

may

be

sent

to

other

C-proc

's

The

C-proc

's

performing

these

tasks

may

be

distributed

either

on

the

same

local

network

or

at

geographically

distant

locations

Intra-site

parallel

crawler

When

all

C-proc

's

run

on

the

same

local

network

and

communicate

through

a

high

speed

interconnect

-LRB-

such

as

LAN

-RRB-

we

call

it

an

intra-site

parallel

crawler

In

Figure

this

scenario

corresponds

to

the

case

where

all

C-proc

's

run

only

on

the

local

network

on

the

top

In

this

case

all

C-proc

's

use

the

same

local

network

when

they

download

pages

from

remote

Web

sites

Therefore

the

network

load

from

C-proc

's

is

centralized

at

a

single

location

where

they

operate

Distributed

crawler

When

C-proc

's

run

at

geographically

distant

locations

connected

by

the

Internet

-LRB-

or

a

wide

area

network

-RRB-

we

call

it

a

distributed

crawler

For

example

one

C-proc

may

run

in

the

US

crawling

all

US

pages

and

another

C-proc

may

run

in

France

crawling

all

European

pages

As

we

discussed

in

the

introduction

a

distributed

crawler

can

disperse

and

even

reduce

the

load

on

the

overall

network

When

C-proc

's

run

at

distant

locations

and

communicate

through

the

Internet

it

becomes

important

how

often

and

how

much

C-proc

's

need

to

communicate

The

bandwidth

between

C-proc

's

may

be

limited

and

For

example

we

may

consider

that

our

proposed

solution

is

a

variation

of

divide

and

conquer

approach

since

we

partition

and

assign

the

Web

to

multiple

processes

C-proc

C-proc

Local

connect

C-proc

collected

pages

queues

of

URLs

to

visit

C-proc

Local

connect

NET

INTER

Figure

General

architecture

of

a

parallel

crawler

S

S

-LRB-

C

-RRB-

-LRB-

C

-RRB-

b

a

c

d

e

f

g

h

i

Figure

Site

S

is

crawled

by

C

and

site

S

is

crawled

by

C

sometimes

unavailable

as

is

often

the

case

with

the

Internet

When

multiple

C-proc

's

download

pages

in

parallel

different

C-proc

's

may

download

the

same

page

multiple

times

In

order

to

avoid

this

overlap

C-proc

's

need

to

coordinate

with

each

other

on

what

pages

to

download

This

coordination

can

be

done

in

one

of

the

following

ways

Independent

At

one

extreme

C-proc

's

may

download

pages

totally

independently

without

any

coordination

That

is

each

C-proc

starts

with

its

own

set

of

seed

URLs

and

follows

links

without

consulting

with

other

C-proc

's

In

this

scenario

downloaded

pages

may

overlap

but

we

may

hope

that

this

overlap

will

not

be

significant

if

all

C-proc

's

start

from

different

seed

URLs

While

this

scheme

has

minimal

coordination

overhead

and

can

be

very

scalable

we

do

not

directly

study

this

option

due

to

its

overlap

problem

Later

we

will

consider

an

improved

version

of

this

option

which

significantly

reduces

overlap

Dynamic

assignment

When

there

exists

a

central

coordinator

that

logically

divides

the

Web

into

small

partitions

-LRB-

using

a

certain

partitioning

function

-RRB-

and

dy-namically

assigns

each

partition

to

a

C-proc

for

download

we

call

it

dynamic

assignment

For

example

assume

that

a

central

coordinator

partitions

the

Web

by

the

site

name

of

a

URL

That

is

pages

in

the

same

site

-LRB-

e.g.

html

and

-RRB-

belong

to

the

same

partition

while

pages

in

different

sites

belong

to

different

partitions

Then

during

a

crawl

the

central

coordinator

constantly

decides

on

what

partition

to

crawl

next

-LRB-

e.g.

the

site

cnn.com

-RRB-

and

sends

URLs

within

this

partition

-LRB-

that

have

been

discovered

so

far

-RRB-

to

a

C-proc

as

seed

URLs

Given

this

request

the

C-proc

downloads

the

pages

and

extracts

links

from

them

When

the

extracted

links

point

to

pages

in

the

same

partition

-LRB-

e.g.

-RRB-

the

C-proc

follows

the

links

but

if

a

link

points

to

a

page

in

another

partition

-LRB-

e.g.

index.html

-RRB-

the

C-proc

reports

the

link

to

the

central

coordinator

The

central

coordinator

later

uses

this

link

as

a

seed

URL

for

the

appropriate

partition

Note

that

the

Web

can

be

partitioned

at

various

gran-ularities

At

one

extreme

the

central

coordinator

may

consider

every

page

as

a

separate

partition

and

assign

individual

URLs

to

C-proc

's

for

download

In

this

case

a

C-proc

does

not

follow

links

because

different

pages

belong

to

separate

partitions

It

simply

reports

all

extracted

URLs

back

to

the

coordinator

Therefore

the

communication

between

a

C-proc

and

the

central

coordinator

may

vary

dramatically

depending

on

the

granularity

of

the

partitioning

function

Static

assignment

When

the

Web

is

partitioned

and

assigned

to

each

C-proc

before

they

start

to

crawl

we

call

it

static

assignment

In

this

case

every

C-proc

knows

which

C-proc

is

responsible

for

which

page

during

a

crawl

and

the

crawler

does

not

need

a

central

coordinator

We

will

shortly

discuss

in

more

detail

how

C-proc

's

operate

under

this

scheme

In

this

paper

we

mainly

focus

on

static

assignment

because

of

its

simplicity

and

scalability

and

defer

the

study

of

dynamic

assignment

to

future

work

Note

that

in

dynamic

assignment

the

central

coordinator

may

become

the

major

bottleneck

because

it

has

to

maintain

a

large

number

of

URLs

reported

from

all

C-proc

's

and

has

to

constantly

coordinate

all

C-proc

's

Thus

the

coordinator

itself

may

also

need

to

be

parallelized

Under

static

assignment

each

C-proc

is

responsible

for

a

certain

partition

of

the

Web

and

has

to

download

pages

within

the

partition

However

some

pages

in

the

partition

may

have

links

to

pages

in

another

partition

We

refer

to

this

type

of

link

as

an

inter-partition

link

To

illustrate

how

a

C-proc

may

handle

inter-partition

links

we

use

Figure

as

our

example

In

the

figure

we

assume

two

C-proc

's

C

and

C

are

responsible

for

sites

S

and

S

respectively

For

now

we

assume

that

the

Web

is

partitioned

by

sites

and

that

the

Web

has

only

S

and

S

Also

we

assume

that

each

C-proc

starts

its

crawl

from

the

root

page

of

each

site

a

and

f.

Firewall

mode

In

this

mode

each

C-proc

downloads

only

the

pages

within

its

partition

and

does

not

follow

any

inter-partition

link

All

inter-partition

links

are

ignored

and

thrown

away

For

example

the

links

a

g

c

g

and

h

d

in

Figure

are

ignored

and

thrown

away

by

C

and

C

In

this

mode

the

overall

crawler

does

not

have

any

overlap

in

the

downloaded

pages

because

a

page

can

be

downloaded

by

only

one

C-proc

if

ever

However

the

overall

crawler

may

not

download

all

pages

that

it

has

to

download

because

some

pages

may

be

reachable

only

through

inter-partition

links

For

example

in

Figure

C

can

download

a

b

and

c

but

not

d

and

e

because

they

can

be

reached

only

through

h

d

link

However

C-proc

's

can

run

quite

independently

in

this

mode

because

they

do

not

conduct

any

run-time

coordination

or

URL

exchanges

Cross-over

mode

Primarily

each

C-proc

downloads

pages

within

its

partition

but

when

it

runs

out

of

pages

in

its

partition

it

also

follows

inter-partition

links

For

example

consider

C

in

Figure

Process

C

first

downloads

pages

a

b

and

c

by

following

links

from

a

At

this

point

C

runs

out

of

pages

in

S

so

it

follows

a

link

to

g

and

starts

exploring

S

After

downloading

g

and

h

it

discovers

a

link

to

d

in

S

so

it

comes

back

to

S

and

downloads

pages

d

and

e

In

this

mode

downloaded

pages

may

clearly

overlap

-LRB-

pages

g

and

h

are

downloaded

twice

-RRB-

but

the

overall

crawler

can

download

more

pages

than

the

firewall

mode

-LRB-

C

downloads

d

and

e

in

this

mode

-RRB-

Also

as

in

the

firewall

mode

C-proc

's

do

not

need

to

communicate

with

each

other

because

they

follow

only

the

links

discovered

by

themselves

Exchange

mode

When

C-proc

's

periodically

and

incrementally

exchange

inter-partition

URLs

we

say

that

they

operate

in

an

exchange

mode

Processes

do

not

follow

inter-partition

links

For

example

C

in

Figure

informs

C

of

page

g

after

it

downloads

page

a

-LRB-

and

c

-RRB-

and

C

transfers

the

URL

of

page

d

to

C

after

it

downloads

page

h.

Note

that

C

does

not

follow

links

to

page

g

It

only

transfers

the

links

to

C

so

that

C

can

download

the

page

In

this

way

the

overall

crawler

can

avoid

overlap

while

maximizing

coverage

Note

that

the

firewall

and

the

cross-over

modes

give

C-proc

's

much

independence

-LRB-

C-proc

's

do

not

need

to

communicate

with

each

other

-RRB-

but

they

may

download

the

same

page

multiple

times

or

may

not

download

some

pages

In

contrast

the

exchange

mode

avoids

these

problems

but

requires

constant

URL

exchange

between

C-proc

's

URL

exchange

minimization

To

reduce

URL

exchange

a

crawler

based

on

the

exchange

mode

may

use

some

of

the

following

techniques

Batch

communication

Instead

of

transferring

an

inter-partition

URL

immediately

after

it

is

discovered

a

C-proc

may

wait

for

a

while

to

collect

a

set

of

URLs

and

send

them

in

a

batch

That

is

with

batching

a

C-proc

collects

all

inter-partition

URLs

until

it

downloads

k

pages

Then

it

partitions

the

collected

URLs

and

sends

them

to

an

appropriate

C-proc

Once

these

URLs

are

transferred

the

C-proc

then

purges

them

and

starts

to

collect

a

new

set

of

URLs

from

the

next

downloaded

pages

Note

that

a

C-proc

does

not

maintain

the

list

of

all

inter-partition

URLs

discovered

so

far

It

only

maintains

the

list

of

inter-partition

links

in

the

current

batch

in

order

to

minimize

the

memory

overhead

for

URL

storage

This

batch

communication

has

various

advantages

over

incremental

communication

First

it

incurs

less

communication

overhead

because

a

set

of

URLs

can

be

sent

in

a

batch

instead

of

sending

one

URL

per

message

Second

the

absolute

number

of

exchanged

URLs

will

also

decrease

For

example

consider

C

in

Figure

The

link

to

page

g

appears

twice

in

page

a

and

in

page

c.

Therefore

if

C

transfers

the

link

to

g

after

downloading

page

a

it

needs

to

send

the

same

URL

again

after

downloading

page

c.

In

contrast

if

C

waits

until

page

c

and

sends

URLs

in

batch

it

needs

to

send

the

URL

for

g

only

once

Replication

It

is

known

that

the

number

of

incoming

links

to

pages

on

the

Web

follows

a

Zipfian

distribution

-LSB-

-RSB-

That

is

a

small

number

of

Web

pages

have

an

extremely

large

number

of

links

pointing

to

them

while

a

majority

of

pages

have

only

a

small

number

of

incoming

links

Thus

we

may

significantly

reduce

URL

exchanges

if

we

replicate

the

most

popular

URLs

at

each

C-proc

-LRB-

by

most

popular

we

mean

the

URLs

with

most

incoming

links

-RRB-

and

stop

transferring

them

between

C-proc

's

That

is

before

we

start

crawling

pages

we

identify

the

most

popular

k

URLs

based

on

the

image

of

the

Web

collected

in

a

previous

crawl

Then

we

replicate

these

URLs

at

each

C-proc

so

that

the

C-proc

's

do

not

exchange

them

during

a

crawl

Since

a

small

number

of

Web

pages

have

a

large

number

of

incoming

links

this

scheme

may

significantly

reduce

URL

exchanges

between

C-proc

's

even

if

we

replicate

a

small

number

of

URLs

Note

that

some

of

the

replicated

URLs

may

be

used

as

the

seed

URLs

for

a

C-proc

That

is

if

some

URLs

in

the

replicated

set

belong

to

the

same

partition

that

a

C-proc

is

responsible

for

the

C-proc

may

use

those

URLs

as

its

seeds

rather

than

starting

from

other

pages

Also

note

that

it

is

possible

that

each

C-proc

tries

to

discover

popular

URLs

on

the

fly

during

a

crawl

instead

of

identifying

them

based

on

the

previous

image

For

example

each

C-proc

may

keep

a

cache

of

recently

seen

URL

entries

This

cache

may

pick

up

popular

URLs

automatically

because

the

popular

URLs

show

up

repeatedly

However

we

believe

that

the

popular

URLs

from

a

previous

crawl

will

be

a

good

approximation

for

the

popular

URLs

in

the

current

Web

Most

popular

Web

pages

-LRB-

such

as

Yahoo!

-RRB-

maintain

their

popularity

for

a

relatively

long

period

of

time

even

if

their

exact

popularity

may

change

slightly

Partitioning

function

So

far

we

have

mainly

assumed

that

the

Web

pages

are

partitioned

by

Web

sites

Clearly

there

exists

a

multitude

of

ways

to

partition

the

Web

including

the

following

URL-hash

based

Based

on

the

hash

value

of

the

URL

of

a

page

we

assign

the

page

to

a

C-proc

In

When

it

downloads

page

c

it

does

not

remember

whether

the

link

to

g

has

been

already

sent

this

scheme

pages

in

the

same

site

can

be

assigned

to

different

C-proc

's

Therefore

the

locality

of

link

structure

is

not

reflected

in

the

partition

and

there

will

be

many

inter-partition

links

Site-hash

based

Instead

of

computing

the

hash

value

on

an

entire

URL

we

compute

the

hash

value

only

on

the

site

name

of

a

URL

-LRB-

e.g.

cnn.com

in

http

cnn.com/index.html

-RRB-

and

assign

the

page

to

a

C-proc

In

this

scheme

note

that

the

pages

in

the

same

site

will

be

allocated

to

the

same

partition

Therefore

only

some

of

the

inter-site

links

will

be

inter-partition

links

and

thus

we

can

reduce

the

number

of

inter-partition

links

quite

significantly

compared

to

the

URL-hash

based

scheme

Hierarchical

Instead

of

using

a

hash-value

we

may

partition

the

Web

hierarchically

based

on

the

URLs

of

pages

For

example

we

may

divide

the

Web

into

three

partitions

-LRB-

the

pages

in

the

com

domain

net

domain

and

all

other

pages

-RRB-

and

allocate

them

to

three

C-proc

's

Even

further

we

may

decompose

the

Web

by

language

or

country

-LRB-

e.g.

mx

for

Mexico

-RRB-

Because

pages

hosted

in

the

same

domain

or

country

may

be

more

likely

to

link

to

pages

in

the

same

domain

scheme

may

have

even

fewer

inter-partition

links

than

the

site-hash

based

scheme

In

this

paper

we

do

not

consider

the

URL-hash

based

scheme

because

it

generates

a

large

number

of

inter-partition

links

When

the

crawler

uses

URL-hash

based

scheme

C-proc

's

need

to

exchange

much

larger

number

of

URLs

-LRB-

exchange

mode

-RRB-

and

the

coverage

of

the

overall

crawler

can

be

much

lower

-LRB-

firewall

mode

-RRB-

In

addition

in

our

later

experiments

we

will

mainly

use

the

site-hash

based

scheme

as

our

partitioning

function

We

chose

this

option

because

it

is

much

simpler

to

implement

and

because

it

captures

the

core

issues

that

we

want

to

study

For

example

under

the

hierarchical

scheme

it

is

not

easy

to

divide

the

Web

into

equal

size

partitions

while

it

is

relatively

straightforward

under

the

site-hash

based

scheme

Also

we

believe

we

can

interpret

the

results

from

the

site-hash

based

scheme

as

the

upper/lower

bound

for

the

hierarchical

scheme

For

instance

assuming

Web

pages

link

to

more

pages

in

the

same

domain

the

number

of

inter-partition

links

will

be

lower

in

the

hierarchical

scheme

than

in

the

site-hash

based

scheme

-LRB-

although

we

could

not

confirm

this

trend

in

our

experiments

-RRB-

In

Figure

we

summarize

the

options

that

we

have

discussed

so

far

The

right-hand

table

in

the

figure

shows

more

detailed

view

on

the

static

coordination

scheme

In

the

diagram

we

highlight

the

main

focus

of

our

paper

with

dark

grey

That

is

we

mainly

study

the

static

coordination

scheme

-LRB-

the

third

column

in

the

left-hand

table

-RRB-

and

we

use

the

site-hash

based

partitioning

for

our

experiments

-LRB-

the

second

row

in

the

second

table

-RRB-

However

during

our

discussion

we

will

also

briefly

explore

the

implications

of

other

According

to

our

experiments

about

of

the

links

in

a

page

point

to

pages

in

the

same

site

on

average

While

the

sizes

of

individual

Web

sites

vary

the

sizes

of

partitions

are

similar

because

each

partition

contains

many

Web

sites

and

their

average

sizes

are

similar

among

partitions

Type

URL-hash

Site-hash

Hierarchical

Distributed

Intra-site

Independent

Dynamic

Static

Batch

Replication

Main

focus

Also

discussed

Partitioning

Exchange

Coordination

Firewall

Cross-over

Mode

Not

discussed

further

Figure

Summary

of

the

options

discussed

options

For

instance

the

firewall

mode

is

an

improved

version

of

the

independent

coordination

scheme

-LRB-

in

the

first

table

-RRB-

so

our

study

on

the

firewall

mode

will

show

the

implications

of

the

independent

coordination

scheme

Also

we

roughly

estimate

the

performance

of

the

URL-hash

based

scheme

-LRB-

first

row

in

the

second

table

-RRB-

when

we

discuss

the

results

from

the

site-hash

based

scheme

Given

our

table

of

crawler

design

space

it

would

be

very

interesting

to

see

what

options

existing

search

engines

selected

for

their

own

crawlers

Unfortunately

this

information

is

impossible

to

obtain

in

most

cases

because

companies

consider

their

technologies

proprietary

and

want

to

keep

them

secret

The

only

two

crawler

designs

that

we

know

of

are

the

prototype

Google

crawler

-LSB-

-RSB-

-LRB-

when

it

was

developed

at

Stanford

-RRB-

and

the

Mercator

crawler

-LSB-

-RSB-

at

Compaq

The

prototype

google

crawler

used

the

intra-site

static

and

site-hash

based

scheme

and

ran

in

exchange

mode

-LSB-

-RSB-

The

Mercator

crawler

uses

the

site-based

hashing

scheme

In

this

section

we

define

metrics

that

will

let

us

quantify

the

advantages

or

disadvantages

of

different

parallel

crawling

schemes

These

metrics

will

be

used

later

in

our

experiments

Overlap

When

multiple

C-proc

's

are

downloading

Web

pages

simultaneously

it

is

possible

that

different

C-proc

's

download

the

same

page

multiple

times

Multiple

downloads

of

the

same

page

are

clearly

undesirable

More

precisely

we

define

the

overlap

of

downloaded

pages

as

N-I

I

Here

N

represents

the

total

number

of

pages

downloaded

by

the

overall

crawler

and

I

represents

the

number

of

unique

pages

downloaded

again

by

the

overall

crawler

Thus

the

goal

of

a

parallel

crawler

is

to

minimize

the

overlap

Note

that

a

parallel

crawler

does

not

have

an

overlap

problem

if

it

is

based

on

the

firewall

mode

-LRB-

Section

Item

-RRB-

or

the

exchange

mode

-LRB-

Section

Item

-RRB-

In

these

modes

a

C-proc

downloads

pages

only

within

its

own

partition

so

the

overlap

is

always

zero

Coverage

When

multiple

C-proc

's

run

independently

it

is

possible

that

they

may

not

download

all

pages

that

they

have

to

In

particular

a

crawler

based

on

the

firewall

mode

-LRB-

Section

Item

-RRB-

may

have

this

problem

because

its

C-proc

's

do

not

follow

inter-partition

links

nor

exchange

the

links

with

others

To

formalize

this

notion

we

define

the

coverage

of

downloaded

pages

as

I

U

where

U

represents

the

total

number

of

pages

that

the

overall

crawler

has

to

download

and

I

is

the

number

of

unique

pages

downloaded

by

the

overall

crawler

For

example

in

Figure

if

C

downloaded

pages

a

b

and

c

and

if

C

downloaded

pages

f

through

i

the

coverage

of

the

overall

crawler

is

because

it

downloaded

pages

out

of

Quality

Often

crawlers

can

not

download

the

whole

Web

and

thus

they

try

to

download

an

important

or

relevant

section

of

the

Web

For

example

a

crawler

may

have

storage

space

only

for

million

pages

and

may

want

to

download

the

most

important

million

pages

To

implement

this

policy

a

crawler

needs

a

notion

of

importance

of

pages

often

called

an

importance

metric

-LSB-

-RSB-

For

example

let

us

assume

that

the

crawler

uses

backlink

count

as

its

importance

metric

That

is

the

crawler

considers

a

page

p

important

when

a

lot

of

other

pages

point

to

it

Then

the

goal

of

the

crawler

is

to

download

the

most

highly-linked

million

pages

To

achieve

this

goal

a

single-process

crawler

may

use

the

following

method

-LSB-

-RSB-

The

crawler

constantly

keeps

track

of

how

many

backlinks

each

page

has

from

the

pages

that

it

has

already

downloaded

and

first

visits

the

page

with

the

highest

backlink

count

Clearly

the

pages

downloaded

in

this

way

may

not

be

the

top

million

pages

because

the

page

selection

is

not

based

on

the

entire

Web

only

on

what

has

been

seen

so

far

Thus

we

may

formalize

the

notion

of

quality

of

downloaded

pages

as

follows

-LSB-

-RSB-

First

we

assume

a

hypothetical

oracle

crawler

which

knows

the

exact

importance

of

every

page

under

a

certain

importance

metric

We

assume

that

the

oracle

crawler

downloads

the

most

important

N

pages

in

total

and

use

P

N

to

represent

that

set

of

N

pages

We

also

use

A

N

to

represent

the

set

of

N

pages

that

an

actual

crawler

would

download

which

would

not

be

necessarily

the

same

as

P

N

Then

we

define

A

N

P

N

P

N

as

the

quality

of

downloaded

pages

by

the

actual

crawler

Under

this

definition

the

quality

represents

the

fraction

of

the

true

top

N

pages

that

are

downloaded

by

the

crawler

Note

that

the

quality

of

a

parallel

crawler

may

be

worse

than

that

of

a

single-process

crawler

because

many

importance

metrics

depend

on

the

global

structure

of

the

Web

-LRB-

e.g.

backlink

count

-RRB-

That

is

each

C-proc

in

a

parallel

crawler

may

know

only

the

pages

that

are

downloaded

by

itself

and

thus

have

less

information

on

page

importance

than

a

single-process

crawler

does

On

the

other

hand

a

single-process

crawler

knows

all

pages

it

has

downloaded

so

far

Therefore

a

C-proc

in

a

parallel

crawler

may

make

a

worse

crawling

decision

than

a

single-process

crawler

In

order

to

avoid

this

quality

problem

C-proc

's

need

to

periodically

exchange

information

on

page

importance

For

example

if

the

backlink

count

is

the

importance

metric

a

C-proc

may

periodically

notify

other

C-proc

's

of

how

many

pages

in

its

partition

have

links

to

pages

in

other

partitions

Note

that

this

backlink

exchange

can

be

naturally

incorporated

in

an

exchange

mode

crawler

-LRB-

Section

Item

-RRB-

In

this

mode

crawling

processes

exchange

inter-partition

URLs

periodically

so

a

C-proc

can

simply

count

how

many

inter-partition

links

it

receives

from

other

C-proc

's

to

count

backlinks

originating

in

other

partitions

More

precisely

if

the

crawler

uses

the

batch

communication

technique

-LRB-

Section

Item

-RRB-

process

C

would

send

a

message

like

-LSB-

com/index.html

-RSB-

to

C

to

notify

that

C

has

seen

links

to

the

page

in

the

current

batch

On

receipt

of

this

message

C

then

increases

the

backlink

count

for

the

page

by

to

reflect

the

inter-partition

links

By

incorporating

this

scheme

we

believe

that

the

quality

of

the

exchange

mode

will

be

better

than

that

of

the

firewall

mode

or

the

cross-over

mode

However

note

that

the

quality

of

an

exchange

mode

crawler

may

vary

depending

on

how

often

it

exchanges

backlink

messages

For

instance

if

crawling

processes

exchange

backlink

messages

after

every

page

download

they

will

have

essentially

the

same

backlink

information

as

a

single-process

crawler

does

-LRB-

They

know

backlink

counts

from

all

pages

that

have

been

downloaded

-RRB-

Therefore

the

quality

of

the

downloaded

pages

would

be

virtually

the

same

as

that

of

a

single-process

crawler

In

contrast

if

C-proc

's

rarely

exchange

backlink

messages

they

do

not

have

accurate

backlink

counts

from

downloaded

pages

so

they

may

make

poor

crawling

decisions

resulting

in

poor

quality

Later

we

will

study

how

often

C-proc

's

should

exchange

backlink

messages

in

order

to

maximize

the

quality

Communication

overhead

The

C-proc

's

in

a

parallel

crawler

need

to

exchange

messages

to

coordinate

their

work

In

particular

C-proc

's

based

on

the

exchange

mode

-LRB-

Section

Item

-RRB-

swap

their

inter-partition

URLs

periodically

To

quantify

how

much

communication

is

required

for

this

exchange

we

define

communication

overhead

as

the

average

number

of

inter-partition

URLs

exchanged

per

downloaded

page

If

the

C-proc

's

send

inter-partition

URLs

incrementally

after

every

page

the

C-proc

's

can

send

the

URL

only

and

other

C-proc

's

can

simply

count

these

URLs

Mode

Coverage

Overlap

Quality

Communication

Firewall

Bad

Good

Bad

Good

Cross-over

Good

Bad

Bad

Good

Exchange

Good

Good

Good

Bad

Table

Comparison

of

three

crawling

modes

For

example

if

a

parallel

crawler

has

downloaded

pages

in

total

and

if

its

C-proc

's

have

exchanged

inter-partition

URLs

its

communication

overhead

is

Note

that

a

crawler

based

on

the

the

firewall

and

the

cross-over

mode

do

not

have

any

communication

overhead

because

they

do

not

exchange

any

inter-partition

URLs

In

Table

we

compare

the

relative

merits

of

the

three

crawling

modes

-LRB-

Section

Items

-RRB-

In

the

table

Good

means

that

the

mode

is

expected

to

perform

relatively

well

for

that

metric

and

Bad

means

that

it

may

perform

worse

compared

to

other

modes

For

instance

the

firewall

mode

does

not

exchange

any

inter-partition

URLs

-LRB-

Communication

Good

-RRB-

and

downloads

pages

only

once

-LRB-

Overlap

Good

-RRB-

but

it

may

not

download

every

page

-LRB-

Coverage

Bad

-RRB-

Also

because

C-proc

's

do

not

exchange

inter-partition

URLs

the

downloaded

pages

may

be

of

lower

quality

than

those

of

an

exchange

mode

crawler

Later

we

will

examine

these

issues

more

quantitatively

through

experiments

based

on

real

Web

data

We

have

discussed

various

issues

related

to

a

parallel

crawler

and

identified

multiple

alternatives

for

its

architecture

In

the

remainder

of

this

paper

we

quantitatively

study

these

issues

through

experiments

conducted

on

real

Web

data

In

all

of

the

following

experiments

we

used

million

Web

pages

in

our

Stanford

WebBase

repository

Because

the

property

of

this

dataset

may

significantly

impact

the

result

of

our

experiments

readers

might

be

interested

in

how

we

collected

these

pages

We

downloaded

the

pages

using

our

Stanford

WebBase

crawler

in

December

in

the

period

of

weeks

In

downloading

the

pages

the

WebBase

crawler

started

with

the

URLs

listed

in

Open

Directory

-LRB-

-RRB-

and

followed

links

We

decided

to

use

the

Open

Directory

URLs

as

seed

URLs

because

these

pages

are

the

ones

that

are

considered

important

by

its

maintainers

In

addition

some

of

our

local

WebBase

users

were

keenly

interested

in

the

Open

Directory

pages

and

explicitly

requested

that

we

cover

them

The

total

number

of

URLs

in

the

Open

Directory

was

around

million

at

that

time

Then

conceptually

the

WebBase

crawler

downloaded

all

these

pages

extracted

URLs

within

the

downloaded

pages

and

followed

links

in

a

breadth-first

manner

-LRB-

The

WebBase

crawler

uses

various

techniques

to

expedite

and

prioritize

crawling

process

but

we

believe

these

optimizations

do

not

affect

the

final

dataset

significantly

-RRB-

Our

dataset

is

relatively

small

-LRB-

million

pages

-RRB-

compared

to

the

full

Web

but

keep

in

mind

that

using

a

significantly

larger

dataset

would

have

made

many

of

our

experiments

prohibitively

expensive

As

we

will

see

each

of

the

graphs

we

present

study

multiple

configurations

and

for

each

configuration

multiple

crawler

runs

were

made

to

Number

of

C-proc

's

Coverage

n

Figure

Number

of

processes

vs.

Coverage

processes

processes

processes

processes

Coverage

Number

of

Seeds

s

Figure

Number

of

seed

URLs

vs.

Coverage

obtain

statistically

valid

data

points

Each

run

involves

simulating

how

one

or

more

C-proc

's

would

visit

the

million

pages

Such

detailed

simulations

are

inherently

very

time

consuming

It

is

clearly

difficult

to

predict

what

would

happen

for

a

larger

dataset

In

the

extended

version

of

this

paper

-LSB-

-RSB-

we

examine

this

data

size

issue

a

bit

more

carefully

and

discuss

whether

a

larger

dataset

would

have

changed

our

conclusions

A

firewall

mode

crawler

-LRB-

Section

Item

-RRB-

has

minimal

communication

overhead

but

it

may

have

coverage

and

quality

problems

-LRB-

Section

-RRB-

In

this

section

we

quantitatively

study

the

effectiveness

of

a

firewall

mode

crawler

using

the

million

pages

in

our

repository

In

particular

we

estimate

the

coverage

-LRB-

Section

Item

-RRB-

of

a

firewall

mode

crawler

when

it

employs

n

C-proc

's

in

parallel

-LRB-

We

discuss

the

quality

issue

of

a

parallel

crawler

later

-RRB-

In

our

experiments

we

considered

the

million

pages

within

our

WebBase

repository

as

the

entire

Web

and

we

used

site-hash

based

partitioning

-LRB-

Section

Item

-RRB-

As

the

seed

URLs

each

C-proc

was

given

random

URLs

from

its

own

partition

so

seed

URLs

were

used

in

total

by

the

overall

crawler

-LRB-

We

discuss

the

effect

of

the

number

of

seed

URLs

shortly

-RRB-

Since

the

crawler

ran

in

firewall

mode

C-proc

's

followed

only

intra-partition

links

not

inter-partition

links

Under

these

settings

we

let

the

C-proc

's

run

until

they

ran

out

of

URLs

After

this

simulated

crawling

we

measured

the

overall

coverage

of

the

crawler

We

performed

these

experiments

with

random

seed

URLS

and

repeated

the

experiments

multiple

times

with

different

seed

URLs

In

all

of

the

runs

the

results

were

essentially

the

same

In

Figure

we

summarize

the

results

from

the

experiments

The

horizontal

axis

represents

n

the

number

of

parallel

C-proc

's

and

the

vertical

axis

shows

the

coverage

of

the

overall

crawler

for

the

given

experiment

Note

that

the

coverage

is

only

even

when

n

-LRB-

a

single-process

-RRB-

This

result

is

because

the

crawler

in

our

experiment

started

with

only

URLs

while

the

actual

dataset

was

collected

with

million

seed

URLs

Thus

some

of

the

million

pages

were

unreachable

from

the

seed

URLs

From

the

figure

it

is

clear

that

the

coverage

decreases

as

the

number

of

processes

increases

This

trend

is

because

the

number

of

inter-partition

links

increases

as

the

Web

is

split

into

smaller

partitions

and

thus

more

pages

are

reachable

only

through

inter-partition

links

From

this

result

we

can

see

that

we

may

run

a

crawler

in

a

firewall

mode

without

much

decrease

in

coverage

with

fewer

than

C-proc

's

For

example

for

the

case

the

coverage

decreases

only

from

the

single-process

case

At

the

same

time

we

can

also

see

that

the

firewall

mode

crawler

becomes

quite

ineffective

with

a

large

number

of

C-proc

's

Less

than

of

the

Web

can

be

downloaded

when

C-proc

's

run

together

each

starting

with

seed

URLs

Clearly

coverage

may

depend

on

the

number

of

seed

URLs

that

each

C-proc

starts

with

To

study

this

issue

we

also

ran

experiments

varying

the

number

of

seed

URLs

s

and

we

show

the

results

in

Figure

The

horizontal

axis

in

the

graph

represents

s

the

total

number

of

seed

URLs

that

the

overall

crawler

used

and

the

vertical

axis

shows

the

coverage

for

that

experiment

For

example

when

s

the

overall

crawler

used

total

seed

URLs

each

C-proc

starting

with

seed

URLs

when

C-proc

's

ran

in

parallel

We

performed

the

experiments

for

C-proc

cases

and

plotted

their

coverage

values

From

this

figure

we

can

observe

the

following

trends

When

a

large

number

of

C-proc

's

run

in

parallel

-LRB-

e.g.

or

-RRB-

the

total

number

of

seed

URLs

affects

the

coverage

very

significantly

For

example

when

processes

run

in

parallel

the

coverage

value

jumps

from

to

if

the

number

of

seed

URLs

increases

from

to

When

only

a

small

number

of

processes

run

in

parallel

-LRB-

e.g.

or

-RRB-

coverage

is

not

significantly

affected

by

the

number

of

seed

URLs

While

coverage

increases

slightly

as

s

increases

the

improvement

is

marginal

Based

on

these

results

we

draw

the

following

conclusions

When

a

relatively

small

number

of

C-proc

's

are

running

in

parallel

a

crawler

using

the

firewall

mode

provides

good

coverage

In

this

case

the

crawler

may

start

with

only

a

small

number

of

seed

URLs

because

coverage

is

not

much

affected

by

the

number

of

seed

URLs

The

firewall

mode

is

not

a

good

choice

if

the

crawler

wants

to

download

every

single

page

on

the

Web

The

crawler

may

miss

some

portion

of

the

Web

particularly

when

it

runs

many

C-proc

's

in

parallel

Example

-LRB-

Generic

search

engine

-RRB-

To

illustrate

how

our

results

could

guide

the

design

of

a

parallel

crawler

consider

the

following

example

Assume

that

to

operate

a

Web

search

engine

we

need

to

download

billion

pages

in

one

Currently

the

Web

is

estimated

to

have

around

billion

pages

overlap

coverage

n

n

number

of

n

's

C-proc

n

n

n

n

Figure

Coverage

vs.

Overlap

for

a

cross-over

mode

crawler

month

Each

machine

that

we

plan

to

run

our

C-proc

's

on

has

Mbps

link

to

the

Internet

and

we

can

use

as

many

machines

as

we

want

Given

that

the

average

size

of

a

web

page

is

around

bytes

we

roughly

need

to

download

bytes

in

one

month

This

download

rate

corresponds

to

Mbps

and

we

need

machines

-LRB-

thus

C-proc

's

-RRB-

to

obtain

the

rate

Given

the

results

of

our

experiment

-LRB-

Figure

-RRB-

we

may

estimate

that

the

coverage

will

be

at

least

with

C-proc

's

Therefore

in

this

scenario

the

firewall

mode

may

be

good

enough

unless

it

is

very

important

to

download

the

entire

Web

Example

-LRB-

High

freshness

-RRB-

As

a

second

example

let

us

now

assume

that

we

have

strong

freshness

requirement

on

the

billion

pages

and

need

to

revisit

every

page

once

every

week

not

once

every

month

This

new

scenario

requires

approximately

Mbps

for

page

download

and

we

need

to

run

C-proc

's

In

this

case

the

coverage

of

the

overall

crawler

decreases

to

less

than

according

to

Figure

Of

course

the

coverage

could

be

larger

than

our

conservative

estimate

but

to

be

safe

one

would

probably

want

to

consider

using

a

crawler

mode

different

than

the

firewall

mode

In

this

section

we

study

the

effectiveness

of

a

cross-over

mode

crawler

-LRB-

Section

Item

-RRB-

A

cross-over

crawler

may

yield

improved

coverage

of

the

Web

since

it

follows

inter-partition

links

when

a

C-proc

runs

out

of

URLs

in

its

own

partition

However

this

mode

incurs

overlap

in

downloaded

pages

-LRB-

Section

Item

-RRB-

because

a

page

can

be

downloaded

by

multiple

C-proc

's

Therefore

the

crawler

increases

its

coverage

at

the

expense

of

overlap

in

the

downloaded

pages

In

Figure

we

show

the

relationship

between

the

coverage

and

the

overlap

of

a

cross-over

mode

crawler

obtained

from

the

following

experiments

We

partitioned

the

pages

using

site-hash

partitioning

and

assigned

them

to

n

C-proc

's

Each

of

the

n

C-proc

's

then

was

given

random

seed

URLs

from

its

partition

and

followed

links

in

the

cross-over

mode

During

this

experiment

we

measured

how

much

overlap

the

overall

crawler

incurred

when

its

coverage

reached

various

points

The

horizontal

axis

in

the

graph

shows

the

coverage

at

a

particular

time

and

the

vertical

axis

shows

the

overlap

at

the

given

coverage

We

performed

the

experiments

for

n

Note

that

in

most

cases

the

overlap

stays

at

zero

until

the

coverage

becomes

relatively

large

For

example

when

n

URL

Hash

Site

Hash

Communication

overhead

n

Number

of

C-proc

's

Figure

Number

of

crawling

processes

vs.

Number

of

URLs

exchanged

per

page

the

overlap

is

zero

until

coverage

reaches

We

can

understand

this

result

by

looking

at

the

graph

in

Figure

According

to

that

graph

a

crawler

with

C-proc

's

can

cover

around

of

the

Web

by

following

only

intra-partition

links

Therefore

even

a

cross-over

mode

crawler

will

follow

only

intra-partition

links

until

its

coverage

reaches

that

point

Only

after

that

each

C-proc

starts

to

follow

inter-partition

links

thus

increasing

the

overlap

For

this

reason

we

believe

that

the

overlap

would

have

been

much

worse

in

the

beginning

of

the

crawl

if

we

adopted

the

independent

model

-LRB-

Section

-RRB-

By

applying

the

partitioning

scheme

to

C-proc

's

we

make

each

C-proc

stay

in

its

own

partition

in

the

beginning

and

suppress

the

overlap

as

long

as

possible

While

the

crawler

in

the

cross-over

mode

is

much

better

than

one

based

on

the

independent

model

it

is

clear

that

the

cross-over

crawler

still

incurs

quite

significant

overlap

For

example

when

C-proc

's

run

in

parallel

in

the

cross-over

mode

the

overlap

becomes

almost

to

obtain

coverage

close

to

For

this

reason

we

do

not

recommend

the

crossover

mode

unless

it

is

absolutely

necessary

to

download

every

page

without

any

communication

between

C-proc

's

To

avoid

the

overlap

and

coverage

problems

an

exchange

mode

crawler

-LRB-

Section

Item

-RRB-

constantly

exchanges

inter-partition

URLs

between

C-proc

's

In

this

section

we

study

the

communication

overhead

-LRB-

Section

Item

-RRB-

of

an

exchange

mode

crawler

and

how

much

we

can

reduce

it

by

replicating

the

most

popular

k

URLs

For

now

let

us

assume

that

a

C-proc

immediately

transfers

inter-partition

URLs

-LRB-

We

will

discuss

batch

communication

later

when

we

discuss

the

quality

of

a

parallel

crawler

-RRB-

In

the

experiments

again

we

split

the

million

pages

into

n

partitions

based

on

site-hash

values

and

ran

n

C-proc

's

in

the

exchange

mode

At

the

end

of

the

crawl

we

measured

how

many

URLs

had

been

exchanged

during

the

crawl

We

show

the

results

in

Figure

In

the

figure

the

horizontal

axis

represents

the

number

of

parallel

C-proc

's

n

and

the

vertical

axis

shows

the

communication

overhead

-LRB-

the

average

number

of

URLs

transferred

per

page

-RRB-

For

comparison

purposes

the

figure

also

shows

the

overhead

for

a

URL-hash

based

scheme

although

the

curve

is

clipped

at

the

top

because

of

its

large

overhead

values

To

explain

the

graph

we

first

note

that

an

average

page

has

out-links

and

about

of

them

point

to

pages

in

the

same

site

Therefore

the

links

are

internally

followed

by

a

C-proc

under

site-hash

partitioning

Only

the

remaining

link

points

to

a

page

in

a

different

site

and

may

be

exchanged

between

processes

Figure

indicates

that

this

URL

exchange

increases

with

the

number

of

processes

For

example

the

C-proc

's

exchanged

URLs

per

page

when

processes

ran

while

they

exchanged

URLs

per

page

when

processes

ran

Based

on

the

graph

we

draw

the

following

conclusions

The

site-hash

based

partitioning

scheme

significantly

reduces

communication

overhead

compared

to

the

URL-hash

based

scheme

We

need

to

transfer

only

up

to

one

link

per

page

-LRB-

or

of

the

links

-RRB-

which

is

significantly

smaller

than

the

URL-hash

based

scheme

For

example

when

we

ran

C-proc

's

using

the

URL-hash

based

scheme

the

crawler

exchanged

links

per

page

under

the

URL-hash

based

scheme

which

was

significantly

larger

than

links

per

page

under

the

site-hash

based

scheme

The

network

bandwidth

used

for

the

URL

exchange

is

relatively

small

compared

to

the

actual

page

download

bandwidth

Under

the

site-hash

based

scheme

at

most

URL

will

be

exchanged

per

page

which

is

about

bytes

Given

that

the

average

size

of

a

Web

page

is

KB

the

URL

exchange

consumes

less

than

of

the

total

network

bandwidth

However

the

overhead

of

the

URL

exchange

on

the

overall

system

can

be

quite

significant

The

processes

need

to

exchange

up

to

one

message

per

page

and

the

message

has

to

go

through

the

TCP/IP

network

stack

at

the

sender

and

the

receiver

Thus

it

is

copied

to

and

from

kernel

space

twice

incurring

two

context

switches

between

the

kernel

and

the

user

mode

Since

these

operations

pose

significant

overhead

even

if

the

message

size

is

small

the

overall

overhead

can

be

important

if

the

processes

exchange

one

message

per

every

downloaded

page

In

the

extended

version

of

this

paper

-LSB-

-RSB-

we

also

study

how

much

we

can

reduce

this

overhead

by

replication

In

short

our

results

indicate

that

we

can

get

significant

reduction

in

communication

cost

-LRB-

between

reduction

-RRB-

when

we

replication

the

most

popular

URLs

in

each

C-proc

When

we

replicated

more

URLs

the

cost

reduction

was

not

as

dramatic

as

the

first

URLs

Thus

we

recommend

replicating

URLs

As

we

discussed

the

quality

-LRB-

Section

Item

-RRB-

of

a

parallel

crawler

can

be

worse

than

that

of

a

single-process

crawler

because

each

C-proc

may

make

crawling

decisions

solely

based

on

the

information

collected

within

its

own

partition

We

now

study

this

quality

issue

In

the

discussion

we

also

study

the

impact

of

the

batch

communication

technique

-LRB-

Section

Item

-RRB-

on

quality

Throughout

the

experiments

in

this

section

we

assume

that

the

crawler

uses

the

number

of

backlinks

to

page

p

as

the

importance

of

p

or

I

-LRB-

p

-RRB-

That

is

if

pages

on

the

In

our

estimation

an

average

URL

was

about

bytes

long

Quality

Number

of

URL

exchanges

Processes

x

-LRB-

a

-RRB-

URL

exchange

vs.

Quality

Number

of

URL

exchanges

Processes

x

Communication

overhead

-LRB-

b

-RRB-

URL

exchange

vs.

Communication

Figure

Crawlers

downloaded

pages

-LRB-

of

-RRB-

Web

have

links

to

page

p

the

importance

of

p

is

I

-LRB-

p

-RRB-

Clearly

there

exist

many

other

ways

to

define

the

importance

of

a

page

but

we

use

this

metric

because

it

-LRB-

or

its

variations

-RRB-

is

being

used

by

some

existing

search

engines

-LSB-

-RSB-

Also

note

that

this

metric

depends

on

the

global

structure

of

the

Web

If

we

use

an

importance

metric

that

solely

depends

on

a

page

itself

not

on

the

global

structure

of

the

Web

the

quality

of

a

parallel

crawler

will

be

essentially

the

same

as

that

of

a

single

crawler

because

each

C-proc

in

a

parallel

crawler

can

make

good

decisions

based

on

the

pages

that

it

has

downloaded

Under

the

backlink

metric

each

C-proc

in

our

experiments

counted

how

many

backlinks

a

page

has

from

the

downloaded

pages

and

visited

the

page

with

the

most

backlinks

first

Remember

that

the

C-proc

's

need

to

periodically

exchange

messages

to

inform

others

of

the

inter-partition

backlinks

Depending

on

how

often

they

exchange

messages

the

quality

of

the

downloaded

pages

will

differ

For

example

if

C-proc

's

never

exchange

messages

the

quality

will

be

the

same

as

that

of

a

firewall

mode

crawler

and

if

they

exchange

messages

after

every

downloaded

page

the

quality

will

be

similar

to

that

of

a

single-process

crawler

To

study

these

issues

we

compared

the

quality

of

the

downloaded

pages

when

C-proc

's

exchanged

backlink

messages

at

various

intervals

and

we

show

the

results

in

Figures

-LRB-

a

-RRB-

-LRB-

a

-RRB-

and

-LRB-

a

-RRB-

Each

graph

shows

the

quality

achieved

by

the

overall

crawler

when

it

downloaded

a

total

of

and

pages

respectively

The

horizontal

axis

in

the

graphs

represents

the

total

number

of

URL

exchanges

during

a

crawl

x

and

the

vertical

axis

shows

the

quality

for

the

given

experiment

For

example

when

x

the

C-proc

's

exchanged

backlink

count

information

only

once

in

the

middle

of

the

crawl

Therefore

the

case

when

x

represents

the

quality

of

a

firewall

mode

crawler

and

the

case

Number

of

URL

exchanges

Quality

Processes

x

-LRB-

a

-RRB-

URL

exchange

vs.

Quality

Communication

overhead

x

Number

of

URL

exchanges

Processes

-LRB-

b

-RRB-

URL

exchange

vs.

Communication

Figure

Crawlers

downloaded

pages

-LRB-

of

-RRB-

when

x

shows

the

quality

of

a

single-process

crawler

In

Figures

-LRB-

b

-RRB-

-LRB-

b

-RRB-

and

-LRB-

b

-RRB-

we

also

show

the

communication

overhead

-LRB-

Section

Item

-RRB-

that

is

the

average

number

of

-LSB-

URL

backlink

count

-RSB-

pairs

exchanged

per

a

downloaded

page

From

these

figures

we

can

observe

the

following

trends

As

the

number

of

crawling

processes

increases

the

quality

of

downloaded

pages

becomes

worse

unless

they

exchange

backlink

messages

often

For

example

in

Figure

-LRB-

a

-RRB-

the

quality

achieved

by

a

crawler

-LRB-

-RRB-

is

significantly

higher

than

that

of

a

crawler

-LRB-

-RRB-

in

the

firewall

mode

-LRB-

x

-RRB-

Again

this

result

is

because

each

C-proc

learns

less

about

the

global

backlink

counts

when

the

Web

is

split

into

smaller

parts

The

quality

of

the

firewall

mode

crawler

-LRB-

x

-RRB-

is

significantly

worse

than

that

of

the

single-process

crawler

-LRB-

x

-RRB-

when

the

crawler

downloads

a

relatively

small

fraction

of

the

pages

-LRB-

Figures

-LRB-

a

-RRB-

and

-LRB-

a

-RRB-

-RRB-

However

the

difference

is

not

very

significant

when

the

crawler

downloads

a

relatively

large

fraction

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

In

other

experiments

when

the

crawler

downloaded

more

than

of

the

pages

the

difference

was

almost

negligible

in

any

case

-LRB-

Due

to

space

limitations

we

do

not

show

the

graphs

-RRB-

Intuitively

this

result

makes

sense

because

quality

is

an

important

issue

only

when

the

crawler

downloads

a

small

portion

of

the

Web

-LRB-

If

the

crawler

will

visit

all

pages

anyway

quality

is

not

relevant

-RRB-

The

communication

overhead

does

not

increase

lin-early

as

the

number

of

URL

exchange

increases

The

graphs

in

Figures

-LRB-

b

-RRB-

-LRB-

b

-RRB-

and

-LRB-

b

-RRB-

are

not

straight

lines

This

result

is

because

a

popular

URL

appears

Number

of

URL

exchanges

Quality

Processes

x

-LRB-

a

-RRB-

URL

exchange

vs.

Quality

Communication

overhead

Processes

x

Number

of

URL

exchanges

-LRB-

b

-RRB-

URL

exchange

vs.

Communication

Figure

Crawlers

downloaded

pages

-LRB-

of

-RRB-

multiple

times

between

backlink

exchanges

Therefore

a

popular

URL

can

be

transferred

as

one

entry

-LRB-

URL

and

its

backlink

count

-RRB-

in

the

exchange

even

if

it

appeared

multiple

times

This

reduction

increases

as

C-proc

's

exchange

backlink

messages

less

frequently

One

does

not

need

a

large

number

of

URL

exchanges

to

achieve

high

quality

Through

multiple

experiments

we

tried

to

identify

how

often

C-proc

's

should

exchange

backlink

messages

to

achieve

the

highest

quality

value

From

these

experiments

we

found

that

a

parallel

crawler

can

get

the

highest

quality

values

even

if

the

processes

communicate

less

than

times

during

a

crawl

We

use

the

following

example

to

illustrate

how

one

can

use

the

results

of

our

experiments

Example

-LRB-

Medium-Scale

Search

Engine

-RRB-

Say

we

plan

to

operate

a

medium-scale

search

engine

and

we

want

to

maintain

about

of

the

Web

-LRB-

M

pages

-RRB-

in

our

index

Our

plan

is

to

refresh

the

index

once

a

month

The

machines

that

we

can

use

have

individual

links

-LRB-

Mbps

-RRB-

to

the

Internet

In

order

to

update

the

index

once

a

month

we

need

about

Mbps

download

bandwidth

so

we

have

to

run

at

least

C-proc

's

on

machines

According

to

Figure

-LRB-

a

-RRB-

-LRB-

download

case

-RRB-

we

can

achieve

the

highest

quality

if

the

C-proc

's

exchange

backlink

messages

times

during

a

crawl

when

processes

run

in

parallel

-LRB-

We

use

the

process

case

because

it

is

the

closest

number

to

-RRB-

Also

from

Figure

-LRB-

b

-RRB-

we

can

see

that

when

C-proc

's

exchange

messages

times

during

a

crawl

they

need

to

exchange

fewer

than

-LSB-

URL

backlink

count

-RSB-

pairs

in

total

Therefore

the

total

network

bandwidth

used

by

the

link

exchange

is

only

-LRB-

-RRB-

-LRB-

-RRB-

of

the

bandwidth

used

by

actual

page

downloads

Also

since

the

exchange

happens

only

times

during

a

crawl

the

context-switch

overhead

for

message

transfers

-LRB-

discussed

in

Section

-RRB-

is

minimal

Note

that

in

this

scenario

we

need

to

exchange

backlink

messages

in

one

month

or

one

message

every

three

days

Therefore

even

if

the

connection

between

C-proc

's

is

unreliable

or

sporadic

we

can

still

use

the

exchange

mode

without

any

problem

Crawlers

are

being

used

more

and

more

often

to

collect

Web

data

for

search

engine

caches

and

data

mining

As

the

size

of

the

Web

grows

it

becomes

increasingly

important

to

use

parallel

crawlers

Unfortunately

almost

nothing

is

known

-LRB-

at

least

in

the

open

literature

-RRB-

about

options

for

parallelizing

crawlers

and

their

performance

Our

paper

addresses

this

shortcoming

by

presenting

several

architectures

and

strategies

for

parallel

crawlers

and

by

studying

their

performance

We

believe

that

our

paper

offers

some

useful

guidelines

for

crawler

designers

helping

them

for

example

select

the

right

number

of

crawling

processes

or

select

the

proper

inter-process

coordination

scheme

In

summary

the

main

conclusions

of

our

study

were

the

following

When

a

small

number

of

crawling

processes

run

in

parallel

-LRB-

in

our

experiment

or

fewer

-RRB-

the

firewall

mode

provides

good

coverage

Given

that

firewall

mode

crawlers

can

run

totally

independently

and

are

easy

to

implement

we

believe

that

it

is

a

good

option

to

consider

The

cases

when

the

firewall

mode

might

not

be

appropriate

are

when

we

need

to

run

more

than

crawling

processes

or

when

we

download

only

a

small

subset

of

the

Web

and

the

quality

of

the

downloaded

pages

is

important

A

crawler

based

on

the

exchange

mode

consumes

small

network

bandwidth

for

URL

exchanges

-LRB-

less

than

of

the

network

bandwidth

-RRB-

It

can

also

minimize

other

overheads

by

adopting

the

batch

communication

technique

In

our

experiments

the

crawler

could

maximize

the

quality

of

the

downloaded

pages

even

if

it

exchanged

backlink

messages

fewer

than

times

during

a

crawl

By

replicating

between

and

popular

URLs

we

can

reduce

the

communication

overhead

by

roughly

Replicating

more

URLs

does

not

significantly

reduce

the

overhead

-LSB-

-RSB-

T.

E.

Anderson

M.

D.

Dahlin

J.

M.

Neefe

D.

A.

Patterson

D.

S.

Roselli

and

R.

Y.

Wang

Serverless

network

file

systems

In

Proc

of

SOSP

-LSB-

-RSB-

A.

Barabasi

and

R.

Albert

Emergence

of

scaling

in

random

networks

Science

-LRB-

-RRB-

-LSB-

-RSB-

A.

Z.

Broder

S.

R.

Kumar

F.

Maghoul

P.

Raghavan

S.

Rajagopalan

R.

Stata

A.

Tomkins

and

J.

L.

Wiener

Graph

structure

in

the

web

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

M.

Burner

Crawling

towards

eterneity

Building

an

archive

of

the

world

wide

web

Web

Techniques

Magazine

-LRB-

-RRB-

May

-LSB-

-RSB-

S.

Chakrabarti

M.

van

den

Berg

and

B.

Dom

Focused

crawling

A

new

approach

to

topic-specific

web

resource

discovery

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

J.

Cho

and

H.

Garcia-Molina

The

evolution

of

the

web

and

implications

for

an

incremental

crawler

In

Proc

of

VLDB

Conf.

-LSB-

-RSB-

J.

Cho

and

H.

Garcia-Molina

Synchronizing

a

database

to

improve

freshness

In

Proc

of

SIGMOD

Conf.

-LSB-

-RSB-

J.

Cho

and

H.

Garcia-Molina

Parallel

crawlers

Technical

report

UCLA

Computer

Science

-LSB-

-RSB-

J.

Cho

H.

Garcia-Molina

and

L.

Page

Efficient

crawling

through

URL

ordering

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

E.

Coffman

Jr.

Z.

Liu

and

R.

R.

Weber

Optimal

robot

scheduling

for

web

search

engines

Technical

report

INRIA

-LSB-

-RSB-

M.

Diligenti

F.

M.

Coetzee

S.

Lawrence

C.

L.

Giles

and

M.

Gori

Focused

crawling

using

context

graphs

In

Proc

of

VLDB

Conf.

-LSB-

-RSB-

D.

Eichmann

The

RBSE

spider

Balancing

effective

search

against

web

load

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

Google

Inc.

-LSB-

-RSB-

A.

Heydon

and

M.

Najork

Mercator

A

scalable

extensible

web

crawler

Word

Wide

Web

-LRB-

-RRB-

December

-LSB-

-RSB-

A.

Heydon

and

M.

Najork

High-performance

web

crawling

Technical

report

SRC

Research

Report

Compaq

Systems

Research

Center

September

-LSB-

-RSB-

D.

Hirschberg

Parallel

algorithms

for

the

transitive

closure

and

the

connected

component

problem

In

Proc

of

STOC

Conf.

-LSB-

-RSB-

M.

Koster

Robots

in

the

web

threat

or

treat

ConneXions

-LRB-

-RRB-

April

-LSB-

-RSB-

O.

A.

McBryan

GENVL

and

WWWW

Tools

for

taming

the

web

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

R.

C.

Miller

and

K.

Bharat

SPHINX

a

framework

for

creating

personal

site-specific

web

crawlers

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

D.

Nassimi

and

S.

Sahni

Parallel

permutation

and

sorting

algorithms

and

a

new

generalized

connection

network

Journal

of

ACM

July

-LSB-

-RSB-

M.

T.

Ozsu

and

P.

Valduriez

Principles

of

Distributed

Database

Systems

Prentice

Hall

-LSB-

-RSB-

L.

Page

and

S.

Brin

The

anatomy

of

a

large-scale

hypertextual

web

search

engine

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

B.

Pinkerton

Finding

what

people

want

Experiences

with

the

web

crawler

In

Proc

of

WWW

Conf.

-LSB-

-RSB-

Robots

exclusion

protocol

-LSB-

-RSB-

A.

S.

Tanenbaum

and

R.

V.

Renesse

Distributed

operating

systems

ACM

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

-RSB-

G.

K.

Zipf

Human

Behaviour

and

the

Principle

of

Least

Effort

an

Introduction

to

Human

Ecology

Addison-Wesley

web crawler
guideline
architecture
parallel crawler
parallelization