Estimating

The

Global

Pagerank

Of

Web

Communities

Localized

search

engines

are

small-scale

systems

that

index

a

particular

community

on

the

web

They

offer

several

benefits

over

their

large-scale

counterparts

in

that

they

are

relatively

inexpensive

to

build

and

can

provide

more

precise

and

complete

search

capability

over

their

relevant

domains

One

disadvantage

such

systems

have

over

large-scale

search

engines

is

the

lack

of

global

PageRank

values

Such

information

is

needed

to

assess

the

value

of

pages

in

the

localized

search

domain

within

the

context

of

the

web

as

a

whole

In

this

paper

we

present

well-motivated

algorithms

to

estimate

the

global

PageRank

values

of

a

local

domain

The

algorithms

are

all

highly

scalable

in

that

given

a

local

domain

of

size

n

they

use

O

-LRB-

n

-RRB-

resources

that

include

computation

time

bandwidth

and

storage

We

test

our

methods

across

a

variety

of

localized

domains

including

site-specific

domains

and

topic-specific

domains

We

demonstrate

that

by

crawling

as

few

as

n

or

additional

pages

our

methods

can

give

excellent

global

PageRank

estimates

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval

G.

-LSB-

Numerical

Analysis

-RSB-

Numerical

Linear

Algebra

G.

-LSB-

Probability

and

Statistics

-RSB-

Markov

Processes

PageRank

Markov

Chain

Stochastic

Complementation

Localized

search

engines

are

small-scale

search

engines

that

index

only

a

single

community

of

the

web

Such

communities

can

be

site-specific

domains

such

as

pages

within

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

pro

t

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

rst

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

speci

c

permission

and/or

a

fee

KDD

August

Philadelphia

Pennsylvania

USA

Copyright

ACM

the

cs.utexas.edu

domain

or

topic-related

communities-for

example

political

websites

Compared

to

the

web

graph

crawled

and

indexed

by

large-scale

search

engines

the

size

of

such

local

communities

is

typically

orders

of

magnitude

smaller

Consequently

the

computational

resources

needed

to

build

such

a

search

engine

are

also

similarly

lighter

By

restricting

themselves

to

smaller

more

manageable

sections

of

the

web

localized

search

engines

can

also

provide

more

precise

and

complete

search

capabilities

over

their

respective

domains

One

drawback

of

localized

indexes

is

the

lack

of

global

information

needed

to

compute

link-based

rankings

The

PageRank

algorithm

-LSB-

-RSB-

has

proven

to

be

an

effective

such

measure

In

general

the

PageRank

of

a

given

page

is

dependent

on

pages

throughout

the

entire

web

graph

In

the

context

of

a

localized

search

engine

if

the

PageRanks

are

computed

using

only

the

local

subgraph

then

we

would

expect

the

resulting

PageRanks

to

reflect

the

perceived

popularity

within

the

local

community

and

not

of

the

web

as

a

whole

For

example

consider

a

localized

search

engine

that

indexes

political

pages

with

conservative

views

A

person

wishing

to

research

the

opinions

on

global

warming

within

the

conservative

political

community

may

encounter

numerous

such

opinions

across

various

websites

If

only

local

PageRank

values

are

available

then

the

search

results

will

reflect

only

strongly

held

beliefs

within

the

community

However

if

global

PageRanks

are

also

available

then

the

results

can

ad-ditionally

reflect

outsiders

views

of

the

conservative

community

-LRB-

those

documents

that

liberals

most

often

access

within

the

conservative

community

-RRB-

Thus

for

many

localized

search

engines

incorporating

global

PageRanks

can

improve

the

quality

of

search

results

However

the

number

of

pages

a

local

search

engine

indexes

is

typically

orders

of

magnitude

smaller

than

the

number

of

pages

indexed

by

their

large-scale

counterparts

Localized

search

engines

do

not

have

the

bandwidth

storage

capacity

or

computational

power

to

crawl

download

and

compute

the

global

PageRanks

of

the

entire

web

In

this

work

we

present

a

method

of

approximating

the

global

PageRanks

of

a

local

domain

while

only

using

resources

of

the

same

order

as

those

needed

to

compute

the

PageRanks

of

the

local

subgraph

Our

proposed

method

looks

for

a

supergraph

of

our

local

subgraph

such

that

the

local

PageRanks

within

this

supergraph

are

close

to

the

true

global

PageRanks

We

construct

this

supergraph

by

iteratively

crawling

global

pages

on

the

current

web

frontier

i.e.

global

pages

with

inlinks

from

pages

that

have

already

been

crawled

In

order

to

provide

Research

Track

Paper

a

good

approximation

to

the

global

PageRanks

care

must

be

taken

when

choosing

which

pages

to

crawl

next

in

this

paper

we

present

a

well-motivated

page

selection

algorithm

that

also

performs

well

empirically

This

algorithm

is

derived

from

a

well-defined

problem

objective

and

has

a

running

time

linear

in

the

number

of

local

nodes

We

experiment

across

several

types

of

local

subgraphs

including

four

topic

related

communities

and

several

site-specific

domains

To

evaluate

performance

we

measure

the

difference

between

the

current

global

PageRank

estimate

and

the

global

PageRank

as

a

function

of

the

number

of

pages

crawled

We

compare

our

algorithm

against

several

heuristics

and

also

against

a

baseline

algorithm

that

chooses

pages

at

random

and

we

show

that

our

method

outperforms

these

other

methods

Finally

we

empirically

demonstrate

that

given

a

local

domain

of

size

n

we

can

provide

good

approximations

to

the

global

PageRank

values

by

crawling

at

most

n

or

additional

pages

The

paper

is

organized

as

follows

Section

gives

an

overview

of

localized

search

engines

and

outlines

their

advantages

over

global

search

Section

provides

background

on

the

PageRank

algorithm

Section

formally

defines

our

problem

and

section

presents

our

page

selection

criteria

and

derives

our

algorithms

Section

provides

experimental

results

section

gives

an

overview

of

related

work

and

finally

conclusions

are

given

in

section

Localized

search

engines

index

a

single

community

of

the

web

typically

either

a

site-specific

community

or

a

topic-specific

community

Localized

search

engines

enjoy

three

major

advantages

over

their

large-scale

counterparts

they

are

relatively

inexpensive

to

build

they

can

offer

more

precise

search

capability

over

their

local

domain

and

they

can

provide

a

more

complete

index

The

resources

needed

to

build

a

global

search

engine

are

enormous

A

study

by

Lyman

et

al.

-LSB-

-RSB-

found

that

the

surface

web

-LRB-

publicly

available

static

sites

-RRB-

consists

of

billion

pages

and

that

the

average

size

of

these

pages

is

approximately

kilobytes

To

download

a

crawl

of

this

size

approximately

terabytes

of

space

is

needed

For

a

researcher

who

wishes

to

build

a

search

engine

with

access

to

a

couple

of

workstations

or

a

small

server

storage

of

this

magnitude

is

simply

not

available

However

building

a

localized

search

engine

over

a

web

community

of

a

hundred

thousand

pages

would

only

require

a

few

gigabytes

of

storage

The

computational

burden

required

to

support

search

queries

over

a

database

this

size

is

more

manageable

as

well

We

note

that

for

topic-specific

search

engines

the

relevant

community

can

be

efficiently

identified

and

downloaded

by

using

a

focused

crawler

-LSB-

-RSB-

For

site-specific

domains

the

local

domain

is

readily

available

on

their

own

web

server

This

obviates

the

need

for

crawling

or

spidering

and

a

complete

and

up-to-date

index

of

the

domain

can

thus

be

guaranteed

This

is

in

contrast

to

their

large-scale

counterparts

which

suffer

from

several

shortcomings

First

crawling

dynamically

generated

pages

pages

in

the

hidden

web

has

been

the

subject

of

research

-LSB-

-RSB-

and

is

a

non-trivial

task

for

an

external

crawler

Second

site-specific

domains

can

enable

the

robots

exclusion

policy

This

prohibits

external

search

engines

crawlers

from

downloading

content

from

the

domain

and

an

external

search

engine

must

instead

rely

on

outside

links

and

anchor

text

to

index

these

restricted

pages

By

restricting

itself

to

only

a

specific

domain

of

the

internet

a

localized

search

engine

can

provide

more

precise

search

results

Consider

the

canonical

ambiguous

search

query

jaguar

which

can

refer

to

either

the

car

manufacturer

or

the

animal

A

scientist

trying

to

research

the

habitat

and

evolutionary

history

of

a

jaguar

may

have

better

success

using

a

finely

tuned

zoology-specific

search

engine

than

querying

Google

with

multiple

keyword

searches

and

wading

through

irrelevant

results

A

method

to

learn

better

ranking

functions

for

retrieval

was

recently

proposed

by

Radlinski

and

Joachims

-LSB-

-RSB-

and

has

been

applied

to

various

local

domains

including

Cornell

University

's

website

-LSB-

-RSB-

The

PageRank

algorithm

defines

the

importance

of

web

pages

by

analyzing

the

underlying

hyperlink

structure

of

a

web

graph

The

algorithm

works

by

building

a

Markov

chain

from

the

link

structure

of

the

web

graph

and

computing

its

stationary

distribution

One

way

to

compute

the

stationary

distribution

of

a

Markov

chain

is

to

find

the

limiting

distribution

of

a

random

walk

over

the

chain

Thus

the

PageRank

algorithm

uses

what

is

sometimes

referred

to

as

the

random

surfer

model

In

each

step

of

the

random

walk

the

surfer

either

follows

an

outlink

from

the

current

page

-LRB-

i.e.

the

current

node

in

the

chain

-RRB-

or

jumps

to

a

random

page

on

the

web

We

now

precisely

define

the

PageRank

problem

Let

U

be

an

m

m

adjacency

matrix

for

a

given

web

graph

such

that

U

ji

if

page

i

links

to

page

j

and

U

ji

otherwise

We

define

the

PageRank

matrix

P

U

to

be

P

U

U

D

U

-LRB-

-RRB-

ve

T

-LRB-

-RRB-

where

D

U

is

the

-LRB-

unique

-RRB-

diagonal

matrix

such

that

U

D

U

is

column

stochastic

is

a

given

scalar

such

that

e

is

the

vector

of

all

ones

and

v

is

a

non-negative

L

normalized

vector

sometimes

called

the

random

surfer

vector

Note

that

the

matrix

D

U

is

well-defined

only

if

each

column

of

U

has

at

least

one

non-zero

entry

i.e.

each

page

in

the

webgraph

has

at

least

one

outlink

In

the

presence

of

such

dangling

nodes

that

have

no

outlinks

one

commonly

used

solution

proposed

by

Brin

et

al.

-LSB-

-RSB-

is

to

replace

each

zero

column

of

U

by

a

non-negative

L

normalized

vector

The

PageRank

vector

r

is

the

dominant

eigenvector

of

the

PageRank

matrix

r

P

U

r

We

will

assume

without

loss

of

generality

that

r

has

an

L

norm

of

one

Computationally

r

can

be

computed

using

the

power

method

This

method

first

chooses

a

random

starting

vector

r

-LRB-

-RRB-

and

iteratively

multiplies

the

current

vector

by

the

PageRank

matrix

P

U

see

Algorithm

In

general

each

iteration

of

the

power

method

can

take

O

-LRB-

m

-RRB-

operations

when

P

U

is

a

dense

matrix

However

in

practice

the

number

of

links

in

a

web

graph

will

be

of

the

order

of

the

number

of

pages

By

exploiting

the

sparsity

of

the

PageRank

matrix

the

work

per

iteration

can

be

reduced

to

O

-LRB-

km

-RRB-

where

k

is

the

average

number

of

links

per

web

page

It

has

also

been

shown

that

the

total

number

of

iterations

needed

for

convergence

is

proportional

to

and

does

not

depend

on

the

size

of

the

web

graph

-LSB-

-RSB-

Finally

the

total

space

needed

is

also

O

-LRB-

km

-RRB-

mainly

to

store

the

matrix

U

Research

Track

Paper

Algorithm

A

linear

time

-LRB-

per

iteration

-RRB-

algorithm

for

computing

PageRank

ComputePR

-LRB-

U

-RRB-

Input

U

Adjacency

matrix

Output

r

PageRank

vector

Choose

-LRB-

randomly

-RRB-

an

initial

non-negative

vector

r

-LRB-

-RRB-

such

that

r

-LRB-

-RRB-

i

repeat

i

i

U

D

U

r

-LRB-

-RRB-

-LCB-

is

the

random

surfing

probability

-RCB-

r

-LRB-

i

-RRB-

-LRB-

-RRB-

v

-LCB-

v

is

the

random

surfer

vector

-RCB-

until

r

-LRB-

i

-RRB-

r

-LRB-

-RRB-

-LCB-

is

the

convergence

threshold

-RCB-

r

r

-LRB-

i

-RRB-

Given

a

local

domain

L

let

G

be

an

N

N

adjacency

matrix

for

the

entire

connected

component

of

the

web

that

contains

L

such

that

G

ji

if

page

i

links

to

page

j

and

G

ji

otherwise

Without

loss

of

generality

we

will

partition

G

as

G

L

G

out

L

out

G

within

-LRB-

-RRB-

where

L

is

the

n

n

local

subgraph

corresponding

to

links

inside

the

local

domain

L

out

is

the

subgraph

that

corresponds

to

links

from

the

local

domain

pointing

out

to

the

global

domain

G

out

is

the

subgraph

containing

links

from

the

global

domain

into

the

local

domain

and

G

within

contains

links

within

the

global

domain

We

assume

that

when

building

a

localized

search

engine

only

pages

inside

the

local

domain

are

crawled

and

the

links

between

these

pages

are

represented

by

the

subgraph

L

The

links

in

L

out

are

also

known

as

these

point

from

crawled

pages

in

the

local

domain

to

uncrawled

pages

in

the

global

domain

As

defined

in

equation

-LRB-

-RRB-

P

G

is

the

PageRank

matrix

formed

from

the

global

graph

G

and

we

define

the

global

PageRank

vector

of

this

graph

to

be

g.

Let

the

n-length

vector

p

be

the

L

normalized

vector

corresponding

to

the

global

PageRank

of

the

pages

in

the

local

domain

L

p

E

L

g

E

L

g

where

E

L

-LSB-

I

-RSB-

is

the

restriction

matrix

that

selects

the

components

from

g

corresponding

to

nodes

in

L.

Let

p

denote

the

PageRank

vector

constructed

from

the

local

domain

subgraph

L

In

practice

the

observed

local

PageRank

p

and

the

global

PageRank

p

will

be

quite

different

One

would

expect

that

as

the

size

of

local

matrix

L

approaches

the

size

of

global

matrix

G

the

global

PageRank

and

the

observed

local

PageRank

will

become

more

similar

Thus

one

approach

to

estimating

the

global

PageRank

is

to

crawl

the

entire

global

domain

compute

its

PageRank

and

extract

the

PageRanks

of

the

local

domain

Typically

however

n

N

i.e.

the

number

of

global

pages

is

much

larger

than

the

number

of

local

pages

Therefore

crawling

all

global

pages

will

quickly

exhaust

all

local

resources

-LRB-

computational

storage

and

bandwidth

-RRB-

available

to

create

the

local

search

engine

We

instead

seek

a

supergraph

F

of

our

local

subgraph

L

with

size

O

-LRB-

n

-RRB-

Our

goal

Algorithm

The

FindGlobalPR

algorithm

FindGlobalPR

-LRB-

L

L

out

T

k

-RRB-

Input

L

zero-one

adjacency

matrix

for

the

local

domain

L

out

zero-one

outlink

matrix

from

L

to

global

subgraph

as

in

-LRB-

-RRB-

T

number

of

iterations

k

number

of

pages

to

crawl

per

iteration

Output

p

an

improved

estimate

of

the

global

PageRank

of

L.

F

L

F

out

L

out

f

ComputePR

-LRB-

F

-RRB-

for

-LRB-

i

to

T

-RRB-

-LCB-

Determine

which

pages

to

crawl

next

-RCB-

pages

SelectNodes

-LRB-

F

F

out

f

k

-RRB-

Crawl

pages

augment

F

and

modify

F

out

-LCB-

Update

PageRanks

for

new

local

domain

-RCB-

f

ComputePR

-LRB-

F

-RRB-

end

-LCB-

Extract

PageRanks

of

original

local

domain

normalize

-RCB-

p

E

L

f

E

L

f

is

to

find

such

a

supergraph

F

with

PageRank

f

so

that

f

when

restricted

to

L

is

close

to

p

Formally

we

seek

to

minimize

GlobalDif

f

-LRB-

f

-RRB-

E

L

f

E

L

f

p

-LRB-

-RRB-

We

choose

the

L

norm

for

measuring

the

error

as

it

does

not

place

excessive

weight

on

outliers

-LRB-

as

the

L

norm

does

for

example

-RRB-

and

also

because

it

is

the

most

commonly

used

distance

measure

in

the

literature

for

comparing

PageRank

vectors

as

well

as

for

detecting

convergence

of

the

algorithm

-LSB-

-RSB-

We

propose

a

greedy

framework

given

in

Algorithm

for

constructing

F

Initially

F

is

set

to

the

local

subgraph

L

and

the

PageRank

f

of

this

graph

is

computed

The

algorithm

then

proceeds

as

follows

First

the

SelectNodes

algorithm

-LRB-

which

we

discuss

in

the

next

section

-RRB-

is

called

and

it

returns

a

set

of

k

nodes

to

crawl

next

from

the

set

of

nodes

in

the

current

crawl

frontier

F

out

These

selected

nodes

are

then

crawled

to

expand

the

local

subgraph

F

and

the

PageRanks

of

this

expanded

graph

are

then

recomputed

These

steps

are

repeated

for

each

of

T

iterations

Finally

the

PageRank

vector

p

which

is

restricted

to

pages

within

the

original

local

domain

is

returned

Given

our

computation

bandwidth

and

memory

restrictions

we

will

assume

that

the

algorithm

will

crawl

at

most

O

-LRB-

n

-RRB-

pages

Since

the

PageRanks

are

computed

in

each

iteration

of

the

algorithm

which

is

an

O

-LRB-

n

-RRB-

operation

we

will

also

assume

that

the

number

of

iterations

T

is

a

constant

Of

course

the

main

challenge

here

is

in

selecting

which

set

of

k

nodes

to

crawl

next

In

the

next

section

we

formally

define

the

problem

and

give

efficient

algorithms

In

this

section

we

present

node

selection

algorithms

that

operate

within

the

greedy

framework

presented

in

the

previous

section

We

first

give

a

well-defined

criteria

for

the

page

selection

problem

and

provide

experimental

evidence

that

this

criteria

can

effectively

identify

pages

that

optimize

our

problem

objective

-LRB-

-RRB-

We

then

present

our

main

Research

Track

Paper

gorithmic

contribution

of

the

paper

a

method

with

linear

running

time

that

is

derived

from

this

page

selection

criteria

Finally

we

give

an

intuitive

analysis

of

our

algorithm

in

terms

of

leaks

and

flows

We

show

that

if

only

the

flow

is

considered

then

the

resulting

method

is

very

similar

to

a

widely

used

page

selection

heuristic

-LSB-

-RSB-

Formulation

For

a

given

page

j

in

the

global

domain

we

define

the

expanded

local

graph

F

j

F

j

F

s

u

T

j

-LRB-

-RRB-

where

u

j

is

the

zero-one

vector

containing

the

outlinks

from

F

into

page

j

and

s

contains

the

inlinks

from

page

j

into

the

local

domain

Note

that

we

do

not

allow

self-links

in

this

framework

In

practice

self-links

are

often

removed

as

they

only

serve

to

inflate

a

given

page

's

PageRank

Observe

that

the

inlinks

into

F

from

node

j

are

not

known

until

after

node

j

is

crawled

Therefore

we

estimate

this

inlink

vector

as

the

expectation

over

inlink

counts

among

the

set

of

already

crawled

pages

s

F

T

e

F

T

e

-LRB-

-RRB-

In

practice

for

any

given

page

this

estimate

may

not

reflect

the

true

inlinks

from

that

page

Furthermore

this

expectation

is

sampled

from

the

set

of

links

within

the

crawled

domain

whereas

a

better

estimate

would

also

use

links

from

the

global

domain

However

the

latter

distribution

is

not

known

to

a

localized

search

engine

and

we

contend

that

the

above

estimate

will

on

average

be

a

better

estimate

than

the

uniform

distribution

for

example

Let

the

PageRank

of

F

be

f

We

express

the

PageRank

f

j

of

the

expanded

local

graph

F

j

as

f

j

-LRB-

x

j

-RRB-

f

j

x

j

-LRB-

-RRB-

where

x

j

is

the

PageRank

of

the

candidate

global

node

j

and

f

j

is

the

L

normalized

PageRank

vector

restricted

to

the

pages

in

F

Since

directly

optimizing

our

problem

goal

requires

knowing

the

global

PageRank

p

we

instead

propose

to

crawl

those

nodes

that

will

have

the

greatest

influence

on

the

PageRanks

of

pages

in

the

original

local

domain

L

influence

-LRB-

j

-RRB-

kL

f

j

-LSB-

k

-RSB-

f

-LSB-

k

-RSB-

-LRB-

-RRB-

E

L

-LRB-

f

j

f

-RRB-

Experimentally

the

influence

score

is

a

very

good

predictor

of

our

problem

objective

-LRB-

-RRB-

For

each

candidate

global

node

j

figure

-LRB-

a

-RRB-

shows

the

objective

function

value

Global

Diff

-LRB-

f

j

-RRB-

as

a

function

of

the

influence

of

page

j

The

local

domain

used

here

is

a

crawl

of

conservative

political

pages

-LRB-

we

will

provide

more

details

about

this

dataset

in

section

-RRB-

we

observed

similar

results

in

other

domains

The

correlation

is

quite

strong

implying

that

the

influence

criteria

can

effectively

identify

pages

that

improve

the

global

PageRank

estimate

As

a

baseline

figure

-LRB-

b

-RRB-

compares

our

objective

with

an

alternative

criteria

outlink

count

The

outlink

count

is

defined

as

the

number

of

outlinks

from

the

local

domain

to

page

j

The

correlation

here

is

much

weaker

Influence

Objective

Outlink

Count

Objective

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

The

correlation

between

our

influence

page

selection

criteria

-LRB-

-RRB-

and

the

actual

objective

function

-LRB-

-RRB-

value

is

quite

strong

-LRB-

b

-RRB-

This

is

in

contrast

to

other

criteria

such

as

outlink

count

which

exhibit

a

much

weaker

correlation

Computation

As

described

for

each

candidate

global

page

j

the

influence

score

-LRB-

-RRB-

must

be

computed

If

f

j

is

computed

exactly

for

each

global

page

j

then

the

PageRank

algorithm

would

need

to

be

run

for

each

of

the

O

-LRB-

n

-RRB-

such

global

pages

j

we

consider

resulting

in

an

O

-LRB-

n

-RRB-

computational

cost

for

the

node

selection

method

Thus

computing

the

exact

value

of

f

j

will

lead

to

a

quadratic

algorithm

and

we

must

instead

turn

to

methods

of

approximating

this

vector

The

algorithm

we

present

works

by

performing

one

power

method

iteration

used

by

the

PageRank

algorithm

-LRB-

Algorithm

-RRB-

The

convergence

rate

for

the

PageRank

algorithm

has

been

shown

to

equal

the

random

surfer

probability

-LSB-

-RSB-

Given

a

starting

vector

x

-LRB-

-RRB-

if

k

PageRank

iterations

are

performed

the

current

PageRank

solution

x

-LRB-

k

-RRB-

satisfies

x

-LRB-

k

-RRB-

x

O

-LRB-

k

x

-LRB-

-RRB-

x

-RRB-

-LRB-

-RRB-

where

x

is

the

desired

PageRank

vector

Therefore

if

only

one

iteration

is

performed

choosing

a

good

starting

vector

is

necessary

to

achieve

an

accurate

approximation

We

partition

the

PageRank

matrix

P

F

j

corresponding

to

the

subgraph

F

j

as

P

F

j

F

s

u

T

j

w

-LRB-

-RRB-

where

F

F

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

-LRB-

-RRB-

e

e

T

s

s

-LRB-

-RRB-

e

u

j

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

u

j

-LRB-

-RRB-

e

w

and

diag

-LRB-

u

j

-RRB-

is

the

diagonal

matrix

with

the

-LRB-

i

i

-RRB-

th

entry

equal

to

one

if

the

i

th

element

of

u

j

equals

one

and

is

zero

otherwise

We

have

assumed

here

that

the

random

surfer

vector

is

the

uniform

vector

and

that

L

has

no

dangling

links

These

assumptions

are

not

necessary

and

serve

only

to

simplify

discussion

and

analysis

A

simple

approach

for

estimating

f

j

is

the

following

First

estimate

the

PageRank

f

j

of

F

j

by

computing

one

PageRank

iteration

over

the

matrix

P

F

j

using

the

starting

vector

f

Then

estimate

f

j

by

removing

the

last

Research

Track

Paper

component

from

our

estimate

of

f

j

-LRB-

i.e.

the

component

corresponding

to

the

added

node

j

-RRB-

and

renormalizing

The

problem

with

this

approach

is

in

the

starting

vector

Recall

from

-LRB-

-RRB-

that

x

j

is

the

PageRank

of

the

added

node

j

The

difference

between

the

actual

PageRank

f

j

of

P

F

j

and

the

starting

vector

is

f

j

x

j

f

-LRB-

x

j

-RRB-

f

j

x

j

f

-LRB-

x

j

-RRB-

f

j

x

j

x

j

j

Thus

by

-LRB-

-RRB-

after

one

PageRank

iteration

we

expect

our

estimate

of

f

j

to

still

have

an

error

of

about

j

In

particular

for

candidate

nodes

j

with

relatively

high

PageRank

x

j

this

method

will

yield

more

inaccurate

results

We

will

next

present

a

method

that

eliminates

this

bias

and

runs

in

O

-LRB-

n

-RRB-

time

Stochastic

Complementation

Since

f

j

as

given

in

-LRB-

-RRB-

is

the

PageRank

of

the

matrix

P

F

j

we

have

f

j

-LRB-

x

j

-RRB-

x

j

F

s

u

T

j

w

f

j

-LRB-

x

j

-RRB-

x

j

F

f

j

-LRB-

x

j

-RRB-

sx

j

u

T

j

f

j

-LRB-

x

j

-RRB-

wx

j

Solving

the

above

system

for

f

j

can

be

shown

to

yield

f

j

-LRB-

F

-LRB-

w

-RRB-

s

u

T

j

-RRB-

f

j

-LRB-

-RRB-

The

matrix

S

F

-LRB-

-RRB-

s

u

T

j

is

known

as

the

stochastic

complement

of

the

column

stochastic

matrix

P

F

j

with

respect

to

the

sub

matrix

F

The

theory

of

stochastic

complementation

is

well

studied

and

it

can

be

shown

the

stochastic

complement

of

an

irreducible

matrix

-LRB-

such

as

the

PageRank

matrix

-RRB-

is

unique

Furthermore

the

stochastic

complement

is

also

irreducible

and

therefore

has

a

unique

stationary

distribution

as

well

For

an

extensive

study

see

-LSB-

-RSB-

It

can

be

easily

shown

that

the

sub-dominant

eigenvalue

of

S

is

at

most

where

is

the

size

of

F

For

sufficiently

large

this

value

will

be

very

close

to

This

is

important

as

other

properties

of

the

PageRank

algorithm

notably

the

algorithm

's

sensitivity

are

dependent

on

this

value

-LSB-

-RSB-

In

this

method

we

estimate

the

length

vector

f

j

by

computing

one

PageRank

iteration

over

the

stochastic

complement

S

starting

at

the

vector

f

f

j

Sf

-LRB-

-RRB-

This

is

in

contrast

to

the

simple

method

outlined

in

the

previous

section

which

first

iterates

over

the

-LRB-

-RRB-

-LRB-

-RRB-

matrix

P

F

j

to

estimate

f

j

and

then

removes

the

last

component

from

the

estimate

and

renormalizes

to

approximate

f

j

The

problem

with

the

latter

method

is

in

the

choice

of

the

-LRB-

-RRB-

length

starting

vector

Consequently

the

PageRank

estimate

given

by

the

simple

method

differs

from

the

true

PageRank

by

at

least

j

where

x

j

is

the

PageRank

of

page

j.

By

using

the

stochastic

complement

we

can

establish

a

tight

lower

bound

of

zero

for

this

difference

To

see

this

consider

the

case

in

which

a

node

k

is

added

to

F

to

form

the

augmented

local

subgraph

F

k

and

that

the

PageRank

of

this

new

graph

is

-LRB-

x

k

-RRB-

f

x

k

Specifi-cally

the

addition

of

page

k

does

not

change

the

PageRanks

of

the

pages

in

F

and

thus

f

k

f

By

construction

of

the

stochastic

complement

f

k

Sf

k

so

the

approximation

given

in

equation

-LRB-

-RRB-

will

yield

the

exact

solution

Next

we

present

the

computational

details

needed

to

efficiently

compute

the

quantity

f

j

f

over

all

known

global

pages

j

We

begin

by

expanding

the

difference

f

j

f

where

the

vector

f

j

is

estimated

as

in

-LRB-

-RRB-

f

j

f

Sf

f

F

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

f

-LRB-

-RRB-

e

e

T

f

-LRB-

w

-RRB-

-LRB-

u

T

j

f

-RRB-

s

f.

-LRB-

-RRB-

Note

that

the

matrix

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

is

diagonal

Letting

o

-LSB-

k

-RSB-

be

the

outlink

count

for

page

k

in

F

we

can

express

the

k

th

diagonal

element

as

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

-LSB-

k

k

-RSB-

o

-LSB-

k

-RSB-

if

u

j

-LSB-

k

-RSB-

o

-LSB-

k

-RSB-

if

u

j

-LSB-

k

-RSB-

Noting

that

-LRB-

o

-LSB-

k

-RSB-

-RRB-

o

-LSB-

k

-RSB-

-LRB-

o

-LSB-

k

-RSB-

-LRB-

o

-LSB-

k

-RSB-

-RRB-

-RRB-

and

rewriting

this

in

matrix

form

yields

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

D

F

D

F

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

diag

-LRB-

u

j

-RRB-

-LRB-

-RRB-

We

use

the

same

identity

to

express

e

e

e

-LRB-

-RRB-

-LRB-

-RRB-

Recall

that

by

definition

we

have

P

F

F

D

F

-LRB-

-RRB-

e

Substituting

-LRB-

-RRB-

and

-LRB-

-RRB-

in

-LRB-

-RRB-

yields

f

j

f

-LRB-

P

F

f

f

-RRB-

F

D

F

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

diag

-LRB-

u

j

-RRB-

f

-LRB-

-RRB-

e

-LRB-

-RRB-

-LRB-

w

-RRB-

-LRB-

u

T

j

f

-RRB-

s

x

y

-LRB-

u

T

j

f

-RRB-

z

-LRB-

-RRB-

noting

that

by

definition

f

P

F

f

and

defining

the

vectors

x

y

and

z

to

be

x

F

D

F

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

diag

-LRB-

u

j

-RRB-

f

-LRB-

-RRB-

y

-LRB-

-RRB-

e

-LRB-

-RRB-

-LRB-

-RRB-

z

-LRB-

w

-RRB-

s.

-LRB-

-RRB-

The

first

term

x

is

a

sparse

vector

and

takes

non-zero

values

only

for

local

pages

k

that

are

siblings

of

the

global

page

j

We

define

-LRB-

i

j

-RRB-

F

if

and

only

if

F

-LSB-

j

i

-RSB-

-LRB-

equiva-lently

page

i

links

to

page

j

-RRB-

and

express

the

value

of

the

component

x

-LSB-

k

-RSB-

as

x

-LSB-

k

-RSB-

k

-LRB-

k

k

-RRB-

F

u

j

-LSB-

k

-RSB-

f

-LSB-

k

-RSB-

o

-LSB-

k

-RSB-

-LRB-

o

-LSB-

k

-RSB-

-RRB-

-LRB-

-RRB-

where

o

-LSB-

k

-RSB-

as

before

is

the

number

of

outlinks

from

page

k

in

the

local

domain

Note

that

the

last

two

terms

y

and

z

are

not

dependent

on

the

current

global

node

j.

Given

the

function

h

j

-LRB-

f

-RRB-

y

-LRB-

u

T

j

f

-RRB-

z

the

quantity

f

j

f

Research

Track

Paper

can

be

expressed

as

f

j

f

k

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

-LRB-

u

T

j

f

-RRB-

z

-LSB-

k

-RSB-

k

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

-LRB-

u

T

j

f

-RRB-

z

-LSB-

k

-RSB-

k

x

-LSB-

k

-RSB-

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

-LRB-

u

T

j

f

-RRB-

z

-LSB-

k

-RSB-

h

j

-LRB-

f

-RRB-

k

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

-LRB-

u

T

j

f

-RRB-

z

-LSB-

k

-RSB-

k

x

-LSB-

k

-RSB-

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

-LRB-

u

T

j

f

-RRB-

z

-LSB-

k

-RSB-

-LRB-

-RRB-

If

we

can

compute

the

function

h

j

in

linear

time

then

we

can

compute

each

value

of

f

j

f

using

an

additional

amount

of

time

that

is

proportional

to

the

number

of

non-zero

components

in

x

These

optimizations

are

carried

out

in

Algorithm

Note

that

-LRB-

-RRB-

computes

the

difference

between

all

components

of

f

and

f

j

whereas

our

node

selection

criteria

given

in

-LRB-

-RRB-

is

restricted

to

the

components

corresponding

to

nodes

in

the

original

local

domain

L.

Let

us

examine

Algorithm

in

more

detail

First

the

algorithm

computes

the

outlink

counts

for

each

page

in

the

local

domain

The

algorithm

then

computes

the

quantity

u

T

j

f

for

each

known

global

page

j

This

inner

product

can

be

written

as

-LRB-

-RRB-

k

-LRB-

k

j

-RRB-

F

out

f

-LSB-

k

-RSB-

o

-LSB-

k

-RSB-

where

the

second

term

sums

over

the

set

of

local

pages

that

link

to

page

j

Since

the

total

number

of

edges

in

F

out

was

assumed

to

have

size

O

-LRB-

-RRB-

-LRB-

recall

that

is

the

number

of

pages

in

F

-RRB-

the

running

time

of

this

step

is

also

O

-LRB-

-RRB-

The

algorithm

then

computes

the

vectors

y

and

z

as

given

in

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

The

L

NormDiff

method

is

called

on

the

components

of

these

vectors

which

correspond

to

the

pages

in

L

and

it

estimates

the

value

of

E

L

-LRB-

y

-LRB-

u

T

j

f

-RRB-

z

-RRB-

for

each

page

j

The

estimation

works

as

follows

First

the

values

of

u

T

j

f

are

discretized

uniformly

into

c

values

-LCB-

a

a

c

-RCB-

The

quantity

E

L

-LRB-

y

a

i

z

-RRB-

is

then

computed

for

each

discretized

value

of

a

i

and

stored

in

a

table

To

evaluate

E

L

-LRB-

y

az

-RRB-

for

some

a

-LSB-

a

a

c

-RSB-

the

closest

discretized

value

a

i

is

determined

and

the

corresponding

entry

in

the

table

is

used

The

total

running

time

for

this

method

is

linear

in

and

the

discretization

parameter

c

-LRB-

which

we

take

to

be

a

constant

-RRB-

We

note

that

if

exact

values

are

desired

we

have

also

developed

an

algorithm

that

runs

in

O

-LRB-

log

-RRB-

time

that

is

not

described

here

In

the

main

loop

we

compute

the

vector

x

as

defined

in

equation

-LRB-

-RRB-

The

nested

loops

iterate

over

the

set

of

pages

in

F

that

are

siblings

of

page

j.

Typically

the

size

of

this

set

is

bounded

by

a

constant

Finally

for

each

page

j

the

scores

vector

is

updated

over

the

set

of

non-zero

components

k

of

the

vector

x

with

k

L

This

set

has

size

equal

to

the

number

of

local

siblings

of

page

j

and

is

a

subset

of

the

total

number

of

siblings

of

page

j.

Thus

each

iteration

of

the

main

loop

takes

constant

time

and

the

total

running

time

of

the

main

loop

is

O

-LRB-

-RRB-

Since

we

have

assumed

that

the

size

of

F

will

not

grow

larger

than

O

-LRB-

n

-RRB-

the

total

running

time

for

the

algorithm

is

O

-LRB-

n

-RRB-

Algorithm

Node

Selection

via

Stochastic

Complementation

SC-Select

-LRB-

F

F

out

f

k

-RRB-

Input

F

zero-one

adjacency

matrix

of

size

corresponding

to

the

current

local

subgraph

F

out

zero-one

outlink

matrix

from

F

to

global

subgraph

f

PageRank

of

F

k

number

of

pages

to

return

Output

pages

set

of

k

pages

to

crawl

next

-LCB-

Compute

outlink

sums

for

local

subgraph

-RCB-

foreach

-LRB-

page

j

F

-RRB-

o

-LSB-

j

-RSB-

k

-LRB-

j

k

-RRB-

F

F

-LSB-

j

k

-RSB-

end

-LCB-

Compute

scalar

u

T

j

f

for

each

global

node

j

-RCB-

foreach

-LRB-

page

j

F

out

-RRB-

g

-LSB-

j

-RSB-

-LRB-

-RRB-

foreach

-LRB-

page

k

-LRB-

k

j

-RRB-

F

out

-RRB-

g

-LSB-

j

-RSB-

g

-LSB-

j

-RSB-

f

-LSB-

k

-RSB-

o

-LSB-

k

-RSB-

end

end

-LCB-

Compute

vectors

y

and

z

as

in

-LRB-

-RRB-

and

-LRB-

-RRB-

-RCB-

y

-LRB-

-RRB-

e

-LRB-

-RRB-

z

-LRB-

w

-RRB-

s

-LCB-

Approximate

y

g

-LSB-

j

-RSB-

z

for

all

values

g

-LSB-

j

-RSB-

-RCB-

norm

diffs

L

NormDiffs

-LRB-

g

E

L

y

E

L

z

-RRB-

foreach

-LRB-

page

j

F

out

-RRB-

-LCB-

Compute

sparse

vector

x

as

in

-LRB-

-RRB-

-RCB-

x

foreach

-LRB-

page

k

-LRB-

k

j

-RRB-

F

out

-RRB-

foreach

-LRB-

page

k

-LRB-

k

k

-RRB-

F

-RRB-

-RRB-

x

-LSB-

k

-RSB-

x

-LSB-

k

-RSB-

f

-LSB-

k

-RSB-

o

-LSB-

k

-RSB-

-LRB-

o

-LSB-

k

-RSB-

-RRB-

end

end

x

x

scores

-LSB-

j

-RSB-

norm

diffs

-LSB-

j

-RSB-

foreach

-LRB-

k

x

-LSB-

k

-RSB-

and

page

k

L

-RRB-

scores

-LSB-

j

-RSB-

scores

-LSB-

j

-RSB-

y

-LSB-

k

-RSB-

g

-LSB-

j

-RSB-

z

-LSB-

k

-RSB-

x

-LSB-

k

-RSB-

y

-LSB-

k

-RSB-

g

-LSB-

j

-RSB-

z

-LSB-

k

-RSB-

-RRB-

end

end

Return

k

pages

with

highest

scores

PageRank

Flows

We

now

present

an

intuitive

analysis

of

the

stochastic

complementation

method

by

decomposing

the

change

in

PageRank

in

terms

of

leaks

and

flows

This

analysis

is

motivated

by

the

decomposition

given

in

-LRB-

-RRB-

PageRank

flow

is

the

increase

in

the

local

PageRanks

originating

from

global

page

j

The

flows

are

represented

by

the

non-negative

vector

-LRB-

u

T

j

f

-RRB-

z

-LRB-

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

-RRB-

The

scalar

u

T

j

f

can

be

thought

of

as

the

total

amount

of

PageRank

flow

that

page

j

has

available

to

distribute

The

vector

z

dictates

how

the

flow

is

allocated

to

the

local

domain

the

flow

that

local

page

k

receives

is

proportional

to

-LRB-

within

a

constant

factor

due

to

the

random

surfer

vector

-RRB-

the

expected

number

of

its

inlinks

The

PageRank

leaks

represent

the

decrease

in

PageRank

resulting

from

the

addition

of

page

j

The

leakage

can

be

quantified

in

terms

of

the

non-positive

vectors

x

and

y

-LRB-

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

-RRB-

For

vector

x

we

can

see

from

equation

-LRB-

-RRB-

that

the

amount

of

PageRank

leaked

by

a

local

page

is

proportional

to

the

weighted

sum

of

the

Research

Track

Paper

Ranks

of

its

siblings

Thus

pages

that

have

siblings

with

higher

PageRanks

-LRB-

and

low

outlink

counts

-RRB-

will

experience

more

leakage

The

leakage

caused

by

y

is

an

artifact

of

the

random

surfer

vector

We

will

next

show

that

if

only

the

flow

term

-LRB-

u

T

j

f

-RRB-

z

is

considered

then

the

resulting

method

is

very

similar

to

a

heuristic

proposed

by

Cho

et

al.

-LSB-

-RSB-

that

has

been

widely

used

for

the

Crawling

Through

URL

Ordering

problem

This

heuristic

is

computationally

cheaper

but

as

we

will

see

later

not

as

effective

as

the

Stochastic

Complementation

method

Our

node

selection

strategy

chooses

global

nodes

that

have

the

largest

influence

-LRB-

equation

-LRB-

-RRB-

-RRB-

If

this

influence

is

approximated

using

only

flows

the

optimal

node

j

is

j

argmax

j

E

L

u

T

j

f

z

argmax

j

u

T

j

f

E

L

z

argmax

j

u

T

j

f

argmax

j

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

u

j

-LRB-

-RRB-

e

f

argmax

j

f

T

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

u

j

The

resulting

page

selection

score

can

be

expressed

as

a

sum

of

the

PageRanks

of

each

local

page

k

that

links

to

j

where

each

PageRank

value

is

normalized

by

o

-LSB-

k

-RSB-

Interestingly

the

normalization

that

arises

in

our

method

differs

from

the

heuristic

given

in

-LSB-

-RSB-

which

normalizes

by

o

-LSB-

k

-RSB-

The

algorithm

PF-Select

which

is

omitted

due

to

lack

of

space

first

computes

the

quantity

f

T

-LRB-

D

F

diag

-LRB-

u

j

-RRB-

-RRB-

u

j

for

each

global

page

j

and

then

returns

the

pages

with

the

k

largest

scores

To

see

that

the

running

time

for

this

algorithm

is

O

-LRB-

n

-RRB-

note

that

the

computation

involved

in

this

method

is

a

subset

of

that

needed

for

the

SC-Select

method

-LRB-

Algorithm

-RRB-

which

was

shown

to

have

a

running

time

of

O

-LRB-

n

-RRB-

In

this

section

we

provide

experimental

evidence

to

verify

the

effectiveness

of

our

algorithms

We

first

outline

our

experimental

methodology

and

then

provide

results

across

a

variety

of

local

domains

Methodology

Given

the

limited

resources

available

at

an

academic

institution

crawling

a

section

of

the

web

that

is

of

the

same

magnitude

as

that

indexed

by

Google

or

Yahoo!

is

clearly

infeasible

Thus

for

a

given

local

domain

we

approximate

the

global

graph

by

crawling

a

local

neighborhood

around

the

domain

that

is

several

orders

of

magnitude

larger

than

the

local

subgraph

Even

though

such

a

graph

is

still

orders

of

magnitude

smaller

than

the

true

global

graph

we

contend

that

even

if

there

exist

some

highly

influential

pages

that

are

very

far

away

from

our

local

domain

it

is

unrealis-tic

for

any

local

node

selection

algorithm

to

find

them

Such

pages

also

tend

to

be

highly

unrelated

to

pages

within

the

local

domain

When

explaining

our

node

selection

strategies

in

section

we

made

the

simplifying

assumption

that

our

local

graph

contained

no

dangling

nodes

This

assumption

was

only

made

to

ease

our

analysis

Our

implementation

efficiently

handles

dangling

links

by

replacing

each

zero

column

of

our

adjacency

matrix

with

the

uniform

vector

We

evaluate

the

algorithm

using

the

two

node

selection

strategies

given

in

Section

and

also

against

the

following

baseline

methods

Random

Nodes

are

chosen

uniformly

at

random

among

the

known

global

nodes

OutlinkCount

Global

nodes

with

the

highest

number

of

outlinks

from

the

local

domain

are

chosen

At

each

iteration

of

the

FindGlobalPR

algorithm

we

evaluate

performance

by

computing

the

difference

between

the

current

PageRank

estimate

of

the

local

domain

E

L

f

E

L

f

and

the

global

PageRank

of

the

local

domain

E

L

g

E

L

g

All

PageRank

calculations

were

performed

using

the

uniform

random

surfer

vector

Across

all

experiments

we

set

the

random

surfer

parameter

to

be

and

used

a

convergence

threshold

of

We

evaluate

the

difference

between

the

local

and

global

PageRank

vectors

using

three

different

metrics

the

L

and

L

norms

and

Kendall

's

tau

The

L

norm

measures

the

sum

of

the

absolute

value

of

the

differences

between

the

two

vectors

and

the

L

norm

measures

the

absolute

value

of

the

largest

difference

Kendall

's

tau

metric

is

a

popular

rank

correlation

measure

used

to

compare

PageRanks

-LSB-

-RSB-

This

metric

can

be

computed

by

counting

the

number

of

pairs

of

pairs

that

agree

in

ranking

and

subtracting

from

that

the

number

of

pairs

of

pairs

that

disagree

in

ranking

The

final

value

is

then

normalized

by

the

total

number

of

n

such

pairs

resulting

in

a

-LSB-

-RSB-

range

where

a

negative

score

signifies

anti-correlation

among

rankings

and

values

near

one

correspond

to

strong

rank

correlation

Results

Our

experiments

are

based

on

two

large

web

crawls

and

were

downloaded

using

the

web

crawler

that

is

part

of

the

Nutch

open

source

search

engine

project

-LSB-

-RSB-

All

crawls

were

restricted

to

only

http

pages

and

to

limit

the

number

of

dynamically

generated

pages

that

we

crawl

we

ig-nored

all

pages

with

urls

containing

any

of

the

characters

or

The

first

crawl

which

we

will

refer

to

as

the

edu

dataset

was

seeded

by

homepages

of

the

top

graduate

computer

science

departments

in

the

USA

as

rated

by

the

US

News

and

World

Report

-LSB-

-RSB-

and

also

by

the

home

pages

of

their

respective

institutions

A

crawl

of

depth

was

performed

restricted

to

pages

within

the

edu

domain

resulting

in

a

graph

with

approximately

million

pages

and

million

links

The

second

crawl

was

seeded

by

the

set

of

pages

under

the

politics

hierarchy

in

the

dmoz

open

directory

project

-LSB-

-RSB-

We

crawled

all

pages

up

to

four

links

away

which

yielded

a

graph

with

million

pages

and

million

links

Within

the

edu

crawl

we

identified

the

five

site-specific

domains

corresponding

to

the

websites

of

the

top

five

graduate

computer

science

departments

as

ranked

by

the

US

News

and

World

Report

This

yielded

local

domains

of

various

sizes

from

-LRB-

UIUC

-RRB-

to

-LRB-

Berkeley

-RRB-

For

each

of

these

site-specific

domains

with

size

n

we

performed

iterations

of

the

FindGlobalPR

algorithm

to

crawl

a

total

of

additional

nodes

Figure

-LRB-

a

-RRB-

gives

the

-LRB-

L

-RRB-

difference

from

the

PageRank

estimate

at

each

iteration

to

the

global

PageRank

for

the

Berkeley

local

domain

The

performance

of

this

dataset

was

representative

of

the

typical

performance

across

the

five

computer

science

site-specific

local

domains

Initially

the

L

difference

between

the

global

and

local

PageRanks

ranged

from

-LRB-

Stanford

-RRB-

to

-LRB-

MIT

-RRB-

For

the

first

several

iterations

the

Research

Track

Paper

Number

of

Iterations

Global

and

Local

PageRank

Difference

-LRB-

-RRB-

Stochastic

Complement

PageRank

Flow

Outlink

Count

Random

Number

of

Iterations

Global

and

Local

PageRank

Difference

-LRB-

-RRB-

Stochastic

Complement

PageRank

Flow

Outlink

Count

Random

Number

of

Iterations

Global

and

Local

PageRank

Difference

-LRB-

-RRB-

Stochastic

Complement

PageRank

Flow

Outlink

Count

Random

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Politics

Figure

L

difference

between

the

estimated

and

true

global

PageRanks

for

-LRB-

a

-RRB-

Berkeley

's

computer

science

website

-LRB-

b

-RRB-

the

site-specific

domain

and

-LRB-

c

-RRB-

the

politics

topic-specific

domain

The

stochastic

complement

method

outperforms

all

other

methods

across

various

domains

three

link-based

methods

all

outperform

the

random

selection

heuristic

After

these

initial

iterations

the

random

heuristic

tended

to

be

more

competitive

with

-LRB-

or

even

outperform

as

in

the

Berkeley

local

domain

-RRB-

the

outlink

count

and

PageRank

flow

heuristics

In

all

tests

the

stochastic

complementation

method

either

outperformed

or

was

competitive

with

the

other

methods

Table

gives

the

average

difference

between

the

final

estimated

global

PageRanks

and

the

true

global

PageRanks

for

various

distance

measures

Algorithm

L

L

Kendall

Stoch

Comp

PR

Flow

Outlink

Random

Table

Average

final

performance

of

various

node

selection

strategies

for

the

five

site-specific

computer

science

local

domains

Note

that

Kendall

's

Tau

measures

similarity

while

the

other

metrics

are

dissimilarity

measures

Stochastic

Complementation

clearly

outperforms

the

other

methods

in

all

metrics

Within

the

politics

dataset

we

also

performed

two

site-specific

tests

for

the

largest

websites

in

the

crawl

org

the

website

for

the

London

based

Adam

Smith

Institute

and

an

online

conservative

journal

As

with

the

edu

local

domains

we

ran

our

algorithm

for

iterations

crawling

a

total

of

nodes

Figure

-LRB-

b

-RRB-

plots

the

results

for

the

domain

In

contrast

to

the

edu

local

domains

the

Random

and

OutlinkCount

methods

were

not

competitive

with

either

the

SC-Select

or

the

PF-Select

methods

Among

all

datasets

and

all

node

selection

methods

the

stochastic

complementation

method

was

most

impressive

in

this

dataset

realizing

a

final

estimate

that

differed

only

from

the

global

PageRank

a

ten-fold

improvement

over

the

initial

local

PageRank

difference

of

For

the

Adam

Smith

local

domain

the

initial

difference

between

the

local

and

global

PageRanks

was

and

the

final

estimates

given

by

the

SC-Select

PF-Select

OutlinkCount

and

Random

methods

were

and

respectively

Within

the

politics

dataset

we

constructed

four

topic-specific

local

domains

The

first

domain

consisted

of

all

pages

in

the

dmoz

politics

category

and

also

all

pages

within

each

of

these

sites

up

to

two

links

away

This

yielded

a

local

domain

of

pages

and

the

results

are

given

in

figure

-LRB-

c

-RRB-

Because

of

the

larger

size

of

the

topic-specific

domains

we

ran

our

algorithm

for

only

iterations

to

crawl

a

total

of

n

nodes

We

also

created

topic-specific

domains

from

three

political

sub-topics

liberalism

conservatism

and

socialism

The

pages

in

these

domains

were

identified

by

their

corresponding

dmoz

categories

For

each

sub-topic

we

set

the

local

domain

to

be

all

pages

within

three

links

from

the

corresponding

dmoz

category

pages

Table

summarizes

the

performance

of

these

three

topic-specific

domains

and

also

the

larger

political

domain

To

quantify

a

global

page

j

's

effect

on

the

global

PageRank

values

of

pages

in

the

local

domain

we

define

page

j

's

impact

to

be

its

PageRank

value

g

-LSB-

j

-RSB-

normalized

by

the

fraction

of

its

outlinks

pointing

to

the

local

domain

impact

-LRB-

j

-RRB-

o

L

-LSB-

j

-RSB-

o

-LSB-

j

-RSB-

g

-LSB-

j

-RSB-

where

o

L

-LSB-

j

-RSB-

is

the

number

of

outlinks

from

page

j

to

pages

in

the

local

domain

L

and

o

-LSB-

j

-RSB-

is

the

total

number

of

j

's

outlinks

In

terms

of

the

random

surfer

model

the

impact

of

page

j

is

the

probability

that

the

random

surfer

-LRB-

-RRB-

is

currently

at

global

page

j

in

her

random

walk

and

-LRB-

-RRB-

takes

an

outlink

to

a

local

page

given

that

she

has

already

decided

not

to

jump

to

a

random

page

For

the

politics

local

domain

we

found

that

many

of

the

pages

with

high

impact

were

in

fact

political

pages

that

should

have

been

included

in

the

dmoz

politics

topic

but

were

not

For

example

the

two

most

influential

global

pages

were

the

political

search

engine

and

the

home

page

of

the

online

political

magazine

com

Among

non-political

pages

the

home

page

of

the

journal

Education

Next

was

most

influential

The

journal

is

freely

available

online

and

contains

articles

regarding

various

aspect

of

education

in

America

To

provide

some

anecdotal

evidence

for

the

effectiveness

of

our

page

selection

methods

we

note

that

the

SC-Select

method

chose

pages

within

the

domain

the

PF-Select

method

discovered

such

pages

while

the

OutlinkCount

and

Random

methods

found

only

pages

each

For

the

conservative

political

local

domain

the

socialist

website

had

a

very

high

impact

score

This

Research

Track

Paper

All

Politics

Algorithm

L

L

Kendall

Stoch

Comp

PR

Flow

Outlink

Random

Conservativism

Algorithm

L

L

Kendall

Stoch

Comp

PR

Flow

Outlink

Random

Liberalism

Algorithm

L

L

Kendall

Stoch

Comp

PR

Flow

Outlink

Random

Socialism

Algorithm

L

L

Kendall

Stoch

Comp

PR

Flow

Outlink

Random

Table

Final

performance

among

node

selection

strategies

for

the

four

political

topic-specific

crawls

Note

that

Kendall

's

Tau

measures

similarity

while

the

other

metrics

are

dissimilarity

measures

was

largely

due

to

a

link

from

the

front

page

of

this

site

to

an

article

regarding

global

warming

published

by

the

National

Center

for

Public

Policy

Research

a

conservative

research

group

in

Washington

DC

Not

surprisingly

the

global

PageRank

of

this

article

-LRB-

which

happens

to

be

on

the

home

page

of

the

NCCPR

-RRB-

was

approximately

whereas

the

local

PageRank

of

this

page

was

only

The

SC-Select

method

yielded

a

global

PageRank

estimate

of

approximately

the

PF-Select

method

estimated

a

value

of

and

the

Random

and

OutlinkCount

methods

yielded

values

of

and

respectively

The

node

selection

framework

we

have

proposed

is

similar

to

the

url

ordering

for

crawling

problem

proposed

by

Cho

et

al.

in

-LSB-

-RSB-

Whereas

our

framework

seeks

to

minimize

the

difference

between

the

global

and

local

PageRank

the

objective

used

in

-LSB-

-RSB-

is

to

crawl

the

most

highly

-LRB-

globally

-RRB-

ranked

pages

first

They

propose

several

node

selection

algorithms

including

the

outlink

count

heuristic

as

well

as

a

variant

of

our

PF-Select

algorithm

which

they

refer

to

as

the

Page-Rank

ordering

metric

They

found

this

method

to

be

most

effective

in

optimizing

their

objective

as

did

a

recent

survey

of

these

methods

by

Baeza-Yates

et

al.

-LSB-

-RSB-

Boldi

et

al.

also

experiment

within

a

similar

crawling

framework

in

-LSB-

-RSB-

but

quantify

their

results

by

comparing

Kendall

's

rank

correlation

between

the

PageRanks

of

the

current

set

of

crawled

pages

and

those

of

the

entire

global

graph

They

found

that

node

selection

strategies

that

crawled

pages

with

the

highest

global

PageRank

first

actually

performed

worse

-LRB-

with

respect

to

Kendall

's

Tau

correlation

between

the

local

and

global

PageRanks

-RRB-

than

basic

depth

first

or

breadth

first

strategies

However

their

experiments

differ

from

our

work

in

that

our

node

selection

algorithms

do

not

use

-LRB-

or

have

access

to

-RRB-

global

PageRank

values

Many

algorithmic

improvements

for

computing

exact

PageRank

values

have

been

proposed

-LSB-

-RSB-

If

such

algorithms

are

used

to

compute

the

global

PageRanks

of

our

local

domain

they

would

all

require

O

-LRB-

N

-RRB-

computation

storage

and

bandwidth

where

N

is

the

size

of

the

global

domain

This

is

in

contrast

to

our

method

which

approximates

the

global

PageRank

and

scales

linearly

with

the

size

of

the

local

domain

Wang

and

Dewitt

-LSB-

-RSB-

propose

a

system

where

the

set

of

web

servers

that

comprise

the

global

domain

communicate

with

each

other

to

compute

their

respective

global

PageRanks

For

a

given

web

server

hosting

n

pages

the

computational

bandwidth

and

storage

requirements

are

also

linear

in

n

One

drawback

of

this

system

is

that

the

number

of

distinct

web

servers

that

comprise

the

global

domain

can

be

very

large

For

example

our

edu

dataset

contains

websites

from

over

different

universities

coordinating

such

a

system

among

a

large

number

of

sites

can

be

very

difficult

Gan

Chen

and

Suel

propose

a

method

for

estimating

the

PageRank

of

a

single

page

-LSB-

-RSB-

which

uses

only

constant

bandwidth

computation

and

space

Their

approach

relies

on

the

availability

of

a

remote

connectivity

server

that

can

supply

the

set

of

inlinks

to

a

given

page

an

assumption

not

used

in

our

framework

They

experimentally

show

that

a

reasonable

estimate

of

the

node

's

PageRank

can

be

obtained

by

visiting

at

most

a

few

hundred

nodes

Using

their

algorithm

for

our

problem

would

require

that

either

the

entire

global

domain

first

be

downloaded

or

a

connectivity

server

be

used

both

of

which

would

lead

to

very

large

web

graphs

The

internet

is

growing

exponentially

and

in

order

to

navigate

such

a

large

repository

as

the

web

global

search

engines

have

established

themselves

as

a

necessity

Along

with

the

ubiquity

of

these

large-scale

search

engines

comes

an

increase

in

search

users

expectations

By

providing

complete

and

isolated

coverage

of

a

particular

web

domain

localized

search

engines

are

an

effective

outlet

to

quickly

locate

content

that

could

otherwise

be

difficult

to

find

In

this

work

we

contend

that

the

use

of

global

PageRank

in

a

localized

search

engine

can

improve

performance

To

estimate

the

global

PageRank

we

have

proposed

an

iterative

node

selection

framework

where

we

select

which

pages

from

the

global

frontier

to

crawl

next

Our

primary

contribution

is

our

stochastic

complementation

page

selection

algorithm

This

method

crawls

nodes

that

will

most

significantly

impact

the

local

domain

and

has

running

time

linear

in

the

number

of

nodes

in

the

local

domain

Experimentally

we

validate

these

methods

across

a

diverse

set

of

local

domains

including

seven

site-specific

domains

and

four

topic-specific

domains

We

conclude

that

by

crawling

an

additional

n

or

pages

our

methods

find

an

estimate

of

the

global

PageRanks

that

is

up

to

ten

times

better

than

just

using

the

local

PageRanks

Furthermore

we

demonstrate

that

our

algorithm

consistently

outperforms

other

existing

heuristics

Research

Track

Paper

Often

times

topic-specific

domains

are

discovered

using

a

focused

web

crawler

which

considers

a

page

's

content

in

conjunction

with

link

anchor

text

to

decide

which

pages

to

crawl

next

-LSB-

-RSB-

Although

such

crawlers

have

proven

to

be

quite

effective

in

discovering

topic-related

content

many

irrelevant

pages

are

also

crawled

in

the

process

Typically

these

pages

are

deleted

and

not

indexed

by

the

localized

search

engine

These

pages

can

of

course

provide

valuable

information

regarding

the

global

PageRank

of

the

local

domain

One

way

to

integrate

these

pages

into

our

framework

is

to

start

the

FindGlobalPR

algorithm

with

the

current

subgraph

F

equal

to

the

set

of

pages

that

were

crawled

by

the

focused

crawler

The

global

PageRank

estimation

framework

along

with

the

node

selection

algorithms

presented

all

require

O

-LRB-

n

-RRB-

computation

per

iteration

and

bandwidth

proportional

to

the

number

of

pages

crawled

T

k

If

the

number

of

iterations

T

is

relatively

small

compared

to

the

number

of

pages

crawled

per

iteration

k

then

the

bottleneck

of

the

algorithm

will

be

the

crawling

phase

However

as

the

number

of

iterations

increases

-LRB-

relative

to

k

-RRB-

the

bottleneck

will

reside

in

the

node

selection

computation

In

this

case

our

algorithms

would

benefit

from

constant

factor

optimizations

Recall

that

the

FindGlobalPR

algorithm

-LRB-

Algorithm

-RRB-

requires

that

the

PageRanks

of

the

current

expanded

local

domain

be

recomputed

in

each

iteration

Recent

work

by

Langville

and

Meyer

-LSB-

-RSB-

gives

an

algorithm

to

quickly

recompute

PageRanks

of

a

given

webgraph

if

a

small

number

of

nodes

are

added

This

algorithm

was

shown

to

give

speedup

of

five

to

ten

times

on

some

datasets

We

plan

to

investigate

this

and

other

such

optimizations

as

future

work

In

this

paper

we

have

objectively

evaluated

our

methods

by

measuring

how

close

our

global

PageRank

estimates

are

to

the

actual

global

PageRanks

To

determine

the

benefit

of

using

global

PageRanks

in

a

localized

search

engine

we

suggest

a

user

study

in

which

users

are

asked

to

rate

the

quality

of

search

results

for

various

search

queries

For

some

queries

only

the

local

PageRanks

are

used

in

ranking

and

for

the

remaining

queries

local

PageRanks

and

the

approximate

global

PageRanks

as

computed

by

our

algorithms

are

used

The

results

of

such

a

study

can

then

be

analyzed

to

determine

the

added

benefit

of

using

the

global

PageRanks

computed

by

our

methods

over

just

using

the

local

PageRanks

Acknowledgements

This

research

was

supported

by

NSF

grant

NSF

Career

Award

and

a

grant

from

Sabre

Inc.

-LSB-

-RSB-

R.

Baeza-Yates

M.

Marin

C.

Castillo

and

A.

Rodriguez

Crawling

a

country

better

strategies

than

breadth-first

for

web

page

ordering

World-Wide

Web

Conference

-LSB-

-RSB-

P.

Boldi

M.

Santini

and

S.

Vigna

Do

your

worst

to

make

the

best

paradoxical

effects

in

pagerank

incremental

computations

Workshop

on

Web

Graphs

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

anatomy

of

a

large-scale

hypertextual

web

search

engine

Computer

Networks

and

ISDN

Systems

-LRB-

-RRB-

-LSB-

-RSB-

S.

Chakrabarti

M.

van

den

Berg

and

B.

Dom

Focused

crawling

a

new

approach

to

topic-specific

web

resource

discovery

World-Wide

Web

Conference

-LSB-

-RSB-

Y.

Chen

Q.

Gan

and

T.

Suel

Local

methods

for

estimating

pagerank

values

Conference

on

Information

and

Knowledge

Management

-LSB-

-RSB-

J.

Cho

H.

Garcia-Molina

and

L.

Page

Efficient

crawling

through

url

ordering

World-Wide

Web

Conference

-LSB-

-RSB-

T.

H.

Haveliwala

and

S.

D.

Kamvar

The

second

eigenvalue

of

the

Google

matrix

Technical

report

Stanford

University

-LSB-

-RSB-

T.

Joachims

F.

Radlinski

L.

Granka

A.

Cheng

C.

Tillekeratne

and

A.

Patel

Learning

retrieval

functions

from

implicit

feedback

-LSB-

-RSB-

S.

D.

Kamvar

T.

H.

Haveliwala

C.

D.

Manning

and

G.

H.

Golub

Exploiting

the

block

structure

of

the

web

for

computing

pagerank

World-Wide

Web

Conference

-LSB-

-RSB-

S.

D.

Kamvar

T.

H.

Haveliwala

C.

D.

Manning

and

G.

H.

Golub

Extrapolation

methods

for

accelerating

pagerank

computation

World-Wide

Web

Conference

-LSB-

-RSB-

A.

N.

Langville

and

C.

D.

Meyer

Deeper

inside

pagerank

Internet

Mathematics

-LSB-

-RSB-

A.

N.

Langville

and

C.

D.

Meyer

Updating

the

stationary

vector

of

an

irreducible

markov

chain

with

an

eye

on

Google

's

pagerank

SIAM

Journal

on

Matrix

Analysis

-LSB-

-RSB-

P.

Lyman

H.

R.

Varian

K.

Swearingen

P.

Charles

N.

Good

L.

L.

Jordan

and

J.

Pal

How

much

information

School

of

Information

Management

and

System

University

of

California

at

Berkely

-LSB-

-RSB-

F.

McSherry

A

uniform

approach

to

accelerated

pagerank

computation

World-Wide

Web

Conference

-LSB-

-RSB-

C.

D.

Meyer

Stochastic

complementation

uncoupling

markov

chains

and

the

theory

of

nearly

reducible

systems

SIAM

Review

-LSB-

-RSB-

US

News

and

World

Report

-LSB-

-RSB-

Dmoz

open

directory

project

-LSB-

-RSB-

Nutch

open

source

search

engine

-LSB-

-RSB-

F.

Radlinski

and

T.

Joachims

Query

chains

learning

to

rank

from

implicit

feedback

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

-LSB-

-RSB-

S.

Raghavan

and

H.

Garcia-Molina

Crawling

the

hidden

web

In

Proceedings

of

the

Twenty-seventh

International

Conference

on

Very

Large

Databases

-LSB-

-RSB-

T.

Tin

Tang

D.

Hawking

N.

Craswell

and

K.

Griffiths

Focused

crawling

for

both

topical

relevance

and

quality

of

medical

information

Conference

on

Information

and

Knowledge

Management

-LSB-

-RSB-

Y.

Wang

and

D.

J.

DeWitt

Computing

pagerank

in

a

distributed

internet

search

system

Proceedings

of

the

VLDB

Conference

Research

Track

Paper

global pagerank
crawling
algorithms
localized search engines
node selection