Lazy

Preservation

Reconstructing

Websites

by

Crawling

the

Crawlers

Backup

of

websites

is

often

not

considered

until

after

a

catastrophic

event

has

occurred

to

either

the

website

or

its

webmaster

We

introduce

lazy

preservation

digital

preservation

performed

as

a

result

of

the

normal

operation

of

web

crawlers

and

caches

Lazy

preservation

is

especially

suitable

for

third

parties

for

example

a

teacher

reconstructing

a

missing

website

used

in

previous

classes

We

evaluate

the

effectiveness

of

lazy

preservation

by

reconstructing

websites

of

varying

sizes

and

composition

using

Warrick

a

web-repository

crawler

Because

of

varying

levels

of

completeness

in

any

one

repository

our

reconstructions

sampled

from

four

different

web

repositories

Google

-LRB-

-RRB-

MSN

-LRB-

-RRB-

Internet

Archive

-LRB-

-RRB-

and

Yahoo

-LRB-

-RRB-

We

also

measured

the

time

required

for

web

resources

to

be

discovered

and

cached

-LRB-

days

-RRB-

as

well

as

how

long

they

remained

in

cache

after

deletion

-LRB-

days

-RRB-

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Online

Information

Services

Web-based

services

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Digital

Libraries

Collection

Measurement

Experimentation

Design

My

old

web

hosting

company

lost

my

site

in

its

entirety

-LRB-

duh

-RRB-

when

a

hard

drive

died

on

them

Needless

to

say

that

I

was

peeved

but

I

do

notice

that

it

is

available

to

browse

on

the

wayback

machine

Does

anyone

have

any

ideas

if

I

can

download

my

full

site

A

request

for

help

at

archive.org

-LSB-

-RSB-

Copyright

Association

for

Computing

Machinery

ACM

acknowledges

that

this

contribution

was

authored

or

co-authored

by

an

employee

contractor

or

affiliate

of

the

U.S.

Government

As

such

the

Government

retains

a

nonexclusive

royalty-free

right

to

publish

or

reproduce

this

article

or

to

allow

others

to

do

so

for

Government

purposes

only

WIDM

November

Arlington

Virginia

USA

Copyright

ACM

Websites

may

be

lost

for

a

number

of

reasons

hard

drive

crashes

file

system

failures

viruses

hacking

etc.

A

lost

website

may

be

restored

if

care

was

taken

to

create

a

backup

beforehand

but

sometimes

webmasters

are

negligent

in

backing

up

their

websites

and

in

cases

such

as

fire

flooding

or

death

of

the

website

owner

backups

are

frequently

unavailable

In

these

cases

webmasters

and

third

parties

may

turn

to

the

Internet

Archive

-LRB-

IA

-RRB-

Wayback

Machine

for

help

According

to

a

representative

from

IA

they

have

performed

over

website

recoveries

in

the

past

year

for

various

individuals

Although

IA

is

often

helpful

it

is

strictly

a

best-effort

approach

that

performs

sporadic

incomplete

and

slow

crawls

of

the

Web

-LRB-

IA

is

at

least

months

out-of-date

-LSB-

-RSB-

-RRB-

Another

source

of

missing

web

content

is

in

the

caches

of

search

engines

-LRB-

SEs

-RRB-

like

Google

MSN

and

Yahoo

that

scour

the

Web

looking

for

content

to

index

Unfortunately

the

SEs

do

not

preserve

canonical

copies

of

all

the

web

resources

they

cache

and

it

is

assumed

that

the

SEs

do

not

keep

web

pages

long

after

they

have

been

removed

from

a

web

server

We

define

lazy

preservation

as

the

collective

digital

preservation

performed

by

web

archives

and

search

engines

on

behalf

of

the

Web

at

large

It

exists

as

a

preservation

service

on

top

of

distributed

incomplete

and

potentially

unreliable

web

repositories

Lazy

preservation

requires

no

individual

effort

or

cost

for

Web

publishers

but

it

also

provides

no

quality

of

service

guarantees

We

explore

the

effectiveness

of

lazy

preservation

by

downloading

websites

of

various

sizes

and

subject

matter

and

reconstructing

them

using

a

web-repository

crawler

named

Warrick

which

recovers

missing

resources

from

four

web

repositories

-LRB-

IA

Google

MSN

and

Yahoo

-RRB-

We

compare

the

downloaded

versions

of

the

sites

with

the

reconstructed

versions

to

measure

how

successful

we

were

at

reconstructing

the

websites

We

also

measure

the

time

it

takes

for

SEs

to

crawl

and

cache

web

pages

that

we

have

created

on

com

and

edu

websites

In

June

we

created

four

synthetic

web

collections

consisting

of

HTML

PDF

and

images

For

days

we

systematically

removed

web

pages

and

measured

how

long

they

remained

cached

by

the

SEs

The

ephemeral

nature

of

the

Web

has

been

widely

ac-knowledged

To

combat

the

disappearance

of

web

resources

Brewster

Kahle

's

Internet

Archive

has

been

archiving

the

Warrick

is

named

after

a

fictional

forensic

scientist

with

a

penchant

for

gambling

Table

Web

repository-supported

data

types

Type

G

Y

M

IA

HTML

C

C

C

C

Plain

text

M

M

M

C

GIF

PNG

JPG

M

M

R

C

JavaScript

M

M

C

MS

Excel

M

S

M

C

MS

PowerPoint

M

M

M

C

MS

Word

M

M

M

C

PDF

M

M

M

C

PostScript

M

S

C

C

Canonical

version

is

stored

M

Modified

version

is

stored

-LRB-

image

thumbnails

or

HTML

conversions

-RRB-

R

Stored

but

not

retrievable

with

direct

URL

S

Indexed

but

stored

version

is

not

accessible

Web

since

-LSB-

-RSB-

National

libraries

are

also

actively

engaged

in

archiving

culturally

important

websites

-LSB-

-RSB-

Systems

like

LOCKSS

-LSB-

-RSB-

have

been

developed

to

ensure

libraries

have

long-term

access

to

publishers

web

content

and

commercial

systems

like

Spurl.net

and

HanzoWeb.com

have

been

developed

to

allow

users

to

archive

selected

web

resources

that

they

deem

important

Other

researchers

have

developed

tools

for

archiving

individual

websites

and

web

pages

InfoMonitor

-LSB-

-RSB-

archives

a

website

's

file

system

and

stores

the

archive

remotely

TTA-pache

-LSB-

-RSB-

is

used

to

archive

requested

pages

from

a

particular

web

server

and

iPROXY

-LSB-

-RSB-

is

used

as

a

proxy

server

to

archive

requested

pages

from

a

variety

of

web

servers

In

many

cases

these

services

can

be

of

some

value

for

recovering

a

lost

website

but

they

are

largely

useless

when

backups

are

inaccessible

or

destroyed

or

when

a

third

party

wants

to

reconstruct

a

website

They

also

require

the

webmaster

to

perform

some

amount

of

work

in

setting

up

configuring

and

monitoring

the

systems

In

regards

to

commercial

search

engines

the

literature

has

mostly

focused

on

measuring

the

amount

of

content

they

have

indexed

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

relevance

of

responses

to

users

queries

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

and

ranking

of

pages

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Lewandowski

et

al.

-LSB-

-RSB-

studied

how

frequently

Google

MSN

and

Yahoo

updated

their

cached

versions

of

web

pages

but

we

are

unaware

of

any

research

that

attempts

to

measure

how

quickly

new

resources

are

added

to

and

removed

from

commercial

SE

caches

or

research

that

explores

the

use

of

SE

caches

for

reconstructing

websites

There

are

many

SEs

and

web

archives

that

index

and

store

Web

content

For

them

to

be

useful

for

website

reconstruction

they

must

at

a

minimum

provide

a

way

to

map

a

given

URL

to

a

stored

resource

To

limit

the

implemen-tation

complexity

we

have

focused

on

what

we

consider

to

be

the

four

most

popular

web

repositories

that

meet

our

minimum

criteria

Recent

measurements

show

that

Google

MSN

and

Yahoo

index

significantly

different

portions

of

the

Web

and

have

an

intersection

of

less

than

-LSB-

-RSB-

Adding

additional

web

repositories

like

ask.com

gigablast.com

in-cywincy

com

and

any

other

web

repository

that

allows

direct

URL

retrieval

would

likely

increase

our

ability

to

reconstruct

websites

t

d

t

a

t

r

t

p

TTL

c

SE

cache

t

m

TTL

ws

t

vulnerable

replicated

endangered

unrecoverable

Web

server

Figure

Timeline

of

SE

resource

acquisition

and

release

Although

SEs

often

publish

index

size

estimates

it

is

difficult

to

estimate

the

number

of

resources

in

each

SE

cache

An

HTML

web

page

may

consist

of

numerous

web

resources

-LRB-

e.g.

images

applets

etc.

-RRB-

that

may

not

be

counted

in

the

estimates

and

not

all

indexed

resources

are

stored

in

the

SE

caches

Google

MSN

and

Yahoo

will

not

cache

an

HTML

page

if

it

contains

a

NOARCHIVE

meta-tag

and

the

http

Cache-control

directives

no-cache

and

no-store

may

also

prevent

caching

of

resources

-LSB-

-RSB-

Only

IA

stores

web

resources

indefinitely

The

SEs

have

proprietary

cache

replacement

and

removal

policies

which

can

only

be

inferred

from

observed

behavior

All

four

web

repositories

perform

sporadic

and

incomplete

crawls

of

websites

making

their

aggregate

performance

important

for

website

reconstruction

Table

shows

the

most

popular

types

of

resources

held

by

the

four

web

repositories

This

table

is

based

on

our

observations

when

reconstructing

websites

with

a

variety

of

content

IA

keeps

a

canonical

version

of

all

web

resources

but

SEs

only

keep

canonical

versions

of

HTML

pages

When

adding

PDF

PostScript

and

Microsoft

Office

-LRB-

Word

Excel

PowerPoint

-RRB-

resources

to

their

cache

the

SEs

create

HTML

versions

of

the

resources

which

are

stripped

of

all

images

SEs

also

keep

only

a

thumbnail

version

of

the

images

they

cache

due

to

copyright

law

MSN

uses

Picsearch

for

their

image

crawling

unfortunately

Picsearch

and

MSN

do

not

support

direct

URL

queries

for

accessing

these

images

so

they

can

not

be

used

for

recovering

website

images

Lifetime

of

a

Web

Resource

Figure

illustrates

the

life

span

of

a

web

resource

from

when

it

is

first

made

available

on

a

web

server

to

when

when

it

is

finally

purged

from

a

SE

cache

A

web

resource

's

time-to

live

on

the

web

server

-LRB-

TTL

ws

-RRB-

is

defined

as

the

number

of

days

from

when

the

resource

is

first

made

accessible

on

the

server

-LRB-

t

-RRB-

to

when

it

is

removed

-LRB-

t

r

-RRB-

A

new

resource

is

vulnerable

until

it

is

discovered

by

a

SE

-LRB-

t

d

-RRB-

and

made

available

in

the

SE

cache

-LRB-

t

a

-RRB-

The

resource

is

replicated

when

it

is

accessible

on

the

web

server

and

in

cache

Once

the

resource

is

removed

from

the

web

server

-LRB-

t

r

-RRB-

it

becomes

endangered

since

it

is

only

accessible

in

cache

When

a

subsequent

crawl

reveals

the

resource

is

no

longer

available

on

the

web

server

-LRB-

t

m

-RRB-

it

will

then

be

purged

from

cache

-LRB-

t

p

-RRB-

and

be

unrecoverable

The

period

between

t

a

and

t

p

define

a

resource

's

time-to-live

in

the

SE

cache

-LRB-

TTL

c

-RRB-

A

resource

is

recoverable

if

it

is

currently

cached

-LRB-

i.e.

is

replicated

or

endangered

-RRB-

A

recoverable

resource

can

only

be

recovered

during

the

TTL

c

period

with

a

probability

of

P

r

the

observed

number

of

days

that

a

resource

is

retrievable

from

the

cache

divided

by

TTL

c

It

should

be

noted

that

the

TTL

ws

and

TTL

c

values

of

a

resource

may

not

necessarily

overlap

A

SE

that

is

trying

to

maximize

the

freshness

of

its

index

will

try

to

minimize

the

difference

between

TTL

ws

and

TTL

c

A

SE

that

is

slow

in

updating

its

index

perhaps

because

it

obtains

crawling

data

from

a

third

party

may

experience

late

caching

where

t

r

t

a

For

a

website

to

be

lazily

preserved

we

would

like

its

resources

to

be

cached

soon

after

their

appearance

on

a

website

-LRB-

have

minimal

vulnerability

-RRB-

SEs

may

also

share

this

goal

if

they

want

to

index

newly

discovered

content

as

quickly

as

possible

Inducing

a

SE

to

crawl

a

website

at

a

specific

time

is

not

currently

possible

Webmasters

may

employ

various

techniques

to

ensure

their

websites

are

crawler-friendly

-LSB-

-RSB-

and

well

connected

to

the

Web

They

may

even

submit

their

website

URLs

to

SEs

or

use

proprietary

mechanisms

like

Google

's

Sitemap

Protocol

-LSB-

-RSB-

but

no

technique

will

guarantee

immediate

indexing

and

caching

of

a

website

We

would

also

like

resources

to

remain

cached

long

after

they

have

been

deleted

from

the

web

server

-LRB-

remain

endangered

-RRB-

so

they

can

be

recovered

for

many

days

after

their

disappearance

SEs

on

the

other

hand

may

want

to

minimize

the

endangered

period

in

order

to

purge

missing

content

from

their

index

Just

as

we

have

no

control

as

to

when

a

SE

crawler

will

visit

we

also

have

no

control

over

cache

eviction

policies

Web

Collection

Design

In

order

to

obtain

measurements

for

TTL

c

and

other

values

in

Figure

we

created

four

synthetic

web

collections

and

placed

them

on

websites

for

which

we

could

obtain

crawling

data

We

deployed

the

collections

in

June

at

four

different

locations

-RRB-

-RRB-

edu

fmccown/lazy

-RRB-

jsmit

and

-RRB-

mln/lazyp

The

com

website

was

new

and

had

never

been

indexed

by

Google

Yahoo

or

MSN

The

edu

websites

had

existed

for

over

a

year

and

had

been

previously

crawled

by

all

three

SEs

In

order

for

the

web

collections

to

be

found

by

the

SEs

we

placed

links

to

the

root

of

each

web

collection

from

the

edu

websites

and

we

submitted

owenbrau

's

base

URL

to

Google

MSN

and

Yahoo

month

prior

to

the

experiment

For

days

we

systematically

removed

resources

from

each

collection

We

examined

the

server

web

logs

to

determine

when

resources

were

crawled

and

we

queried

Google

MSN

and

Yahoo

daily

to

determine

when

the

resources

were

cached

We

organized

each

web

collection

into

a

series

of

update

bins

-LRB-

directories

-RRB-

which

contained

a

number

of

HTML

pages

referencing

the

same

three

inline

images

-LRB-

GIF

JPG

and

PNG

-RRB-

and

a

number

of

PDF

files

An

index.html

file

-LRB-

with

a

single

inline

image

-RRB-

in

the

root

of

the

web

collection

pointed

to

each

of

the

bins

An

index.html

file

in

each

bin

pointed

to

the

HTML

pages

and

PDF

files

so

a

web

crawler

could

easily

find

all

the

resources

All

these

files

were

static

and

did

not

change

throughout

the

day

period

except

the

index.html

files

in

each

bin

which

were

modified

when

links

to

deleted

web

pages

were

removed

In

all

there

were

HTML

files

PDF

files

and

images

in

each

web

collection

More

detail

about

the

organization

of

the

web

collections

and

what

the

pages

and

images

looked

like

can

be

found

in

-LSB-

-RSB-

The

PDF

and

HTML

pages

were

made

to

look

like

typical

web

pages

with

around

words

per

page

The

text

for

each

page

was

randomly

generated

from

a

standard

English

dictionary

By

using

random

words

we

avoided

creating

duplicate

pages

that

a

SE

may

reject

-LSB-

-RSB-

Unfortunately

using

random

words

may

cause

pages

to

be

flagged

as

spam

-LSB-

-RSB-

Each

HTML

and

PDF

page

contained

a

unique

identifier

-LRB-

UID

-RRB-

at

the

top

of

each

page

-LRB-

e.g.

that

included

identifiers

the

web

collection

-LRB-

e.g.

means

the

mln

collection

-RRB-

bin

number

-LRB-

e.g.

means

bin

-RRB-

page

number

and

resource

type

-LRB-

e.g.

means

page

number

from

bin

and

PDF

resource

-RRB-

The

UID

contains

spaces

to

allow

for

more

efficient

querying

of

the

SE

caches

The

TTL

ws

for

each

resource

in

the

web

collection

is

a

function

of

its

bin

number

b

and

page

number

p

TTL

ws

b

-LRB-

p

-RRB-

-LRB-

-RRB-

Daily

SE

Queries

In

designing

our

daily

SE

queries

care

was

taken

to

perform

a

limited

number

of

daily

queries

to

not

overburden

the

SEs

We

could

have

queried

the

SEs

using

the

URL

for

each

resource

but

this

might

have

led

to

our

resources

being

cached

prematurely

it

is

possible

that

if

a

SE

is

queried

for

a

URL

it

did

not

index

that

it

would

add

the

URL

to

a

list

of

URLs

to

be

crawled

at

a

later

date

This

is

how

IA

's

advanced

search

interface

handles

missing

URLs

from

users

queries

To

determine

which

HTML

and

PDF

resources

had

been

cached

we

queried

using

subsets

of

the

resources

UIDs

and

looked

for

cached

URLs

in

the

results

pages

For

example

to

find

PDF

resources

from

the

mln

collection

we

queried

each

SE

to

return

the

top

PDF

results

from

the

site

that

contain

the

exact

phrase

It

is

necessary

to

divulge

the

site

in

the

query

or

multiple

results

from

the

site

will

not

be

returned

Although

this

tells

the

SE

on

which

site

the

resource

is

located

it

does

not

divulge

the

URL

of

the

resource

To

query

for

cached

images

we

queried

for

the

globally

unique

filename

given

to

each

image

Crawling

and

Caching

Observations

Although

the

web

server

logs

registered

visits

from

a

variety

of

crawlers

we

report

only

on

crawls

from

Google

Inktomi

-LRB-

Yahoo

-RRB-

and

MSN

Alexa

Internet

-LRB-

who

provides

crawls

to

IA

-RRB-

only

accessed

our

collection

once

-LRB-

induced

through

our

use

of

the

Alexa

toolbar

-RRB-

A

separate

IA

robot

accessed

less

than

of

the

collections

likely

due

to

several

submissions

we

made

to

their

Wayback

Machine

's

advanced

search

interface

early

in

the

experiment

Further

analysis

of

the

log

data

can

seen

in

a

companion

paper

-LSB-

-RSB-

We

report

only

detailed

measurements

on

HTML

resources

-LRB-

PDF

resources

were

similar

-RRB-

Images

were

crawled

and

cached

far

less

frequently

Google

and

Picsearch

-LRB-

the

MSN

Images

provider

-RRB-

were

the

only

ones

to

crawl

a

significant

number

of

images

The

edu

collections

had

of

their

images

crawled

and

owenbrau

had

of

its

images

crawled

Only

unique

images

appeared

in

Google

Images

all

from

MSN

only

allows

limiting

the

results

page

to

Due

to

a

technical

mishap

beyond

our

control

we

were

unable

to

obtain

crawling

data

for

days

for

owebrau

and

parts

of

days

and

for

the

edu

web

collections

We

were

also

prevented

from

making

cache

queries

on

days

and

Table

Caching

of

HTML

resources

from

web

collections

-LRB-

HTML

resources

in

each

collection

-RRB-

Web

URLs

crawled

URLs

cached

t

ca

T

T

L

c

P

r

Endangered

collection

G

M

Y

G

M

Y

G

M

Y

G

M

Y

G

M

Y

fmccown

jsmit

mln

owenbrau

Ave

Figure

Crawling

-LRB-

top

-RRB-

and

caching

-LRB-

bottom

-RRB-

of

HTML

resources

from

the

mln

web

collection

the

mln

collection

Google

likely

used

an

image

duplication

detection

algorithm

to

prevent

duplicate

images

from

different

URLs

from

being

cached

Only

one

image

-LRB-

from

fmccown

-RRB-

appeared

in

MSN

Images

of

the

cached

images

fell

out

of

cache

during

our

experiment

Table

summarizes

the

performance

of

each

SE

to

crawl

and

cache

HTML

resources

from

each

of

the

four

web

collections

This

table

does

not

include

index.html

resources

which

had

an

infinite

T

T

L

ws

We

believe

there

was

an

error

in

the

MSN

query

script

which

caused

fewer

resources

to

be

found

in

the

MSN

cache

but

the

percentage

of

crawled

URLs

provides

an

upper

bound

on

the

number

of

cached

resources

this

has

little

to

no

effect

on

the

other

measurements

reported

The

three

SEs

showed

equal

desire

to

crawl

HTML

and

PDF

resources

Inktomi

-LRB-

Yahoo

-RRB-

crawled

times

as

many

resources

as

MSN

and

Google

crawled

almost

times

as

many

resources

than

MSN

Google

was

the

only

SE

to

crawl

and

cache

any

resources

from

the

new

owenbrau

website

From

a

preservation

perspective

Google

out-performed

MSN

and

Yahoo

in

nearly

every

category

Google

cached

the

highest

percentage

of

HTML

resources

-LRB-

-RRB-

and

took

only

days

on

average

to

cache

new

resources

from

the

edu

web

collections

On

average

Google

cached

HTML

resources

for

the

longest

period

of

time

-LRB-

days

-RRB-

consistently

provided

access

to

the

cached

resources

-LRB-

-RRB-

and

were

the

slowest

to

remove

cached

resources

that

were

deleted

from

the

web

server

-LRB-

days

-RRB-

Although

Yahoo

cached

more

HTML

resources

and

kept

the

resources

cached

for

a

longer

period

than

MSN

the

probability

of

accessing

a

resource

on

any

given

day

was

only

compared

to

for

MSN

Figure

provides

an

interesting

look

at

the

crawling

and

caching

behavior

of

Google

Yahoo

and

MSN

These

graphs

illustrate

the

crawling

and

caching

of

HTML

resources

from

the

mln

collection

the

other

two

edu

collections

exhibited

similar

behavior

The

resources

are

sorted

by

TTL

ws

with

the

longest-living

resources

appearing

on

the

bottom

The

index.html

files

which

were

never

removed

from

the

web

collection

have

an

infinite

TTL

-LRB-

inf

-RRB-

The

red

diagonal

line

indicates

the

decay

of

the

web

collection

on

any

particular

day

only

resources

below

the

red

line

were

accessible

from

the

web

server

On

the

top

row

of

Figure

blue

dots

indicate

resources

that

were

crawled

on

a

particular

day

When

resources

were

requested

that

had

been

deleted

the

web

server

responded

with

a

-LRB-

not

found

-RRB-

code

represented

by

green

dots

above

the

red

line

The

bottom

row

of

Figure

shows

the

cached

HTML

resources

-LRB-

blue

-RRB-

resulting

from

the

crawls

Some

pages

in

Yahoo

were

indexed

but

not

cached

-LRB-

green

-RRB-

As

Figure

illustrates

both

Google

and

MSN

were

quick

to

make

resources

available

in

their

cache

soon

after

they

were

crawled

and

they

were

quick

to

purge

resources

from

their

cache

when

a

crawl

revealed

the

resources

were

no

longer

available

on

the

web

server

A

surprising

finding

is

that

many

of

the

HTML

resources

that

were

previously

purged

from

Google

's

cache

reappeared

on

day

and

remained

cached

for

the

remainder

of

our

experiment

The

other

two

edu

collections

exhibited

similar

behavior

for

HTML

resources

HTML

and

PDF

resources

from

owenbrau

appeared

in

the

Google

cache

on

day

for

the

first

time

these

resources

had

been

deleted

from

the

web

server

days

before

day

Manual

inspection

weeks

after

the

experiment

had

concluded

revealed

that

the

pages

remained

in

Google

's

cache

and

fell

out

months

later

Yahoo

was

very

sporadic

in

caching

resources

there

was

often

a

lag

time

of

days

between

the

crawl

of

a

resource

and

its

appearance

in

cache

Many

of

the

crawled

resources

never

appeared

in

Yahoo

's

cache

Although

Inktomi

crawled

nearly

every

available

HTML

resource

on

day

only

half

of

those

resources

ever

became

available

in

the

Yahoo

cache

We

have

observed

through

subsequent

interaction

with

Yahoo

that

links

to

cached

content

may

appear

and

disappear

when

performing

the

same

query

just

a

few

seconds

apart

This

likely

accounts

for

the

observed

cache

inconsistency

We

have

observed

from

our

measurements

that

nearly

all

new

HTML

and

PDF

resources

that

we

placed

on

known

websites

were

crawled

and

cached

by

Google

several

days

A

A

D

B

C

E

F

G

B

C

E

F

added

W

W

changed

identical

missing

Figure

Lost

website

-LRB-

left

-RRB-

reconstructed

website

-LRB-

center

-RRB-

and

reconstruction

diagram

-LRB-

right

-RRB-

ter

they

were

discovered

Resources

on

a

new

website

were

not

cached

for

months

Yahoo

and

MSN

were

times

slower

than

Google

to

acquire

new

resources

and

Yahoo

incurs

a

long

transfer

delay

from

Inktomi

's

crawls

into

their

cache

We

have

also

observed

that

cached

resources

are

often

purged

from

all

three

caches

as

soon

as

a

crawl

reveals

the

resources

are

missing

but

in

the

case

of

Google

many

HTML

resources

have

reappeared

weeks

after

being

removed

Images

tend

to

be

largely

ignored

Search

engines

may

crawl

and

cache

other

websites

differ-ently

depending

on

a

variety

of

factors

including

perceived

level

of

importance

-LRB-

e.g.

PageRank

-RRB-

and

modification

rates

Crawling

policies

may

also

be

changed

over

time

This

experiment

merely

provides

a

glimpse

into

the

current

caching

behavior

of

the

top

three

SEs

that

has

not

been

documented

before

Our

findings

suggest

that

SEs

vary

greatly

in

the

level

of

access

they

provide

to

cached

resources

and

that

websites

are

likely

to

be

reconstructed

more

successfully

if

they

are

reconstructed

quickly

after

being

lost

Reconstructions

should

also

be

performed

several

days

in

a

row

to

ensure

maximum

access

to

web

repository

holdings

In

some

cases

it

may

even

be

beneficial

to

attempt

recovering

resources

even

a

month

after

they

have

been

lost

We

define

a

reconstructed

website

to

be

the

collection

of

recovered

resources

that

share

the

same

URIs

as

the

resources

from

a

lost

website

or

from

some

previous

version

of

the

lost

website

-LSB-

-RSB-

The

recovered

resources

may

be

equivalent

to

or

very

different

from

the

lost

resources

For

websites

that

are

composed

of

static

files

recovered

resources

would

be

equivalent

to

the

files

that

were

lost

For

sites

produced

dynamically

using

CGI

PHP

etc.

the

recovered

resources

would

match

the

client

's

view

of

the

resources

and

would

be

useful

to

the

webmaster

in

rebuilding

the

server-side

components

The

server-side

components

are

currently

not

recoverable

using

lazy

preservation

-LRB-

see

Section

-RRB-

To

quantify

the

difference

between

a

reconstructed

website

and

a

lost

website

we

classify

the

recovered

resources

from

the

website

graphs

A

website

can

be

represented

as

a

graph

G

-LRB-

V

E

-RRB-

where

each

resource

r

i

-LRB-

HTML

PDF

image

etc.

-RRB-

identified

by

a

URI

is

a

node

v

i

and

there

exists

a

directed

edge

from

v

i

to

v

j

when

there

is

a

hyperlink

or

reference

from

r

i

to

r

j

The

left

side

of

Figure

shows

a

web

graph

for

some

website

W

if

we

began

to

crawl

it

starting

at

A.

Suppose

W

was

lost

and

reconstructed

forming

the

website

W

represented

in

the

center

of

Figure

For

each

resource

r

i

in

W

we

may

examine

its

corresponding

resource

r

i

in

W

that

shares

the

same

URI

and

categorize

r

i

as

identical

-LRB-

r

i

is

byte-for-byte

identical

to

r

i

-RRB-

changed

-LRB-

r

i

is

not

identical

to

r

i

-RRB-

or

missing

-LRB-

r

i

could

not

be

found

in

any

web

-RRB-

We

would

categorize

those

resources

in

W

that

did

not

share

a

URI

with

any

resource

in

W

as

added

-LRB-

r

i

was

not

a

part

of

the

current

website

but

was

recovered

due

to

a

reference

from

r

j

-RRB-

Figure

shows

that

resources

A

G

and

E

were

reconstructed

and

are

identical

to

their

lost

versions

An

older

version

of

B

was

found

-LRB-

B

-RRB-

that

pointed

to

G

a

resource

that

does

not

currently

exist

in

W

Since

B

does

not

reference

D

we

did

not

know

to

recover

it

-LRB-

it

is

possible

that

G

is

actually

D

renamed

-RRB-

An

older

version

of

C

was

found

and

although

it

still

references

F

F

could

not

be

found

in

any

web

repository

A

measure

of

change

between

the

lost

website

W

and

the

reconstructed

website

W

can

be

described

using

the

following

difference

vector

difference

-LRB-

W

W

-RRB-

R

changed

W

R

missing

W

R

added

W

-LRB-

-RRB-

For

Figure

the

difference

vector

is

-LRB-

-RRB-

-LRB-

-RRB-

The

best

case

scenario

would

be

-LRB-

-RRB-

the

complete

reconstruction

of

a

website

A

completely

unrecoverable

website

would

have

a

difference

vector

of

-LRB-

-RRB-

The

difference

vector

for

a

reconstructed

website

can

be

illustrated

as

a

reconstruction

diagram

as

shown

on

the

right

side

of

Figure

The

changed

identical

and

missing

resources

form

the

core

of

the

reconstructed

website

The

dark

gray

portion

of

the

core

grows

as

the

percentage

of

changed

resource

increases

The

hole

in

the

center

of

the

core

grows

as

the

percentage

of

missing

resources

increases

The

added

resources

appear

as

crust

around

the

core

This

representation

will

be

used

later

in

Table

when

we

report

on

the

websites

we

reconstructed

in

our

experiments

Warrick

Operation

Warrick

our

web-repository

crawler

is

able

to

reconstruct

a

website

when

given

a

base

URL

pointing

to

where

the

site

used

to

exist

The

web

repositories

are

crawled

by

issuing

queries

in

the

form

of

URLs

to

access

their

stored

holdings

For

example

Google

's

cached

version

of

can

be

accessed

like

so

If

Google

has

not

cached

the

page

an

error

page

will

be

generated

Otherwise

the

cached

page

can

be

stripped

of

any

Google-added

HTML

and

the

page

can

be

parsed

for

links

to

other

resources

from

the

foo.edu

domain

-LRB-

and

other

domains

if

necessary

-RRB-

Most

repositories

require

two

or

more

queries

to

obtain

a

resource

For

each

URL

the

file

extension

-LRB-

if

present

-RRB-

is

examined

to

determine

if

the

URL

is

an

image

-LRB-

png

gif

jpg

etc.

-RRB-

or

other

resource

type

All

three

SEs

use

a

different

method

for

retrieving

images

than

for

other

resource

types

IA

has

the

same

interface

regardless

of

the

type

We

would

have

better

accuracy

at

determining

if

a

given

URL

referenced

an

image

or

not

if

we

knew

the

URL

's

resource

MIME

type

but

this

information

is

not

available

to

us

IA

is

the

first

web

repository

queried

by

Warrick

because

it

keeps

a

canonical

version

of

all

web

resources

When

querying

for

an

image

URL

if

IA

does

not

have

the

image

then

Google

and

Yahoo

are

queried

one

at

a

time

until

one

of

them

returns

an

image

Google

and

Yahoo

do

not

publicize

the

cached

date

of

their

images

so

it

is

not

possible

to

pick

the

most

recently

cached

image

Table

Results

of

website

reconstructions

MIME

type

groupings

-LRB-

orig/recovered

-RRB-

Difference

vector

Website

PR

Total

HTML

Images

Other

-LRB-

Changed

Missing

Added

-RRB-

Recon

diag

Almost

identical

New

recon

diag

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

If

a

non-image

resource

is

being

retrieved

again

IA

is

queried

first

If

IA

has

the

resource

and

the

resource

does

not

have

a

MIME

type

of

text/html

then

the

SEs

are

not

queried

since

they

only

store

canonical

versions

of

HTML

resources

If

the

resource

does

have

a

text/html

MIME

type

-LRB-

or

IA

did

not

have

a

copy

-RRB-

then

all

three

SEs

are

queried

the

cache

dates

of

the

resources

are

compared

-LRB-

if

available

-RRB-

and

the

most

recent

resource

is

chosen

Warrick

will

search

HTML

resources

for

URLs

to

other

resources

and

add

them

to

the

crawl

frontier

-LRB-

a

queue

-RRB-

Resources

are

recovered

in

breadth-first

order

and

reconstruction

continues

until

the

frontier

is

empty

All

recovered

resources

are

stored

on

the

local

filesystem

and

a

log

is

kept

of

recovered

and

missing

resources

Warrick

limits

its

requests

per

day

to

the

web

repositories

based

on

their

published

API

values

-LRB-

Google

Yahoo

MSN

-RRB-

or

lacking

an

API

our

best

guess

-LRB-

IA

-RRB-

If

any

repository

's

limit

is

exceeded

Warrick

will

checkpoint

and

sleep

for

hours

Reconstruction

Experiment

and

Results

To

gauge

the

effectiveness

of

lazy

preservation

for

website

reconstruction

we

compared

the

snap-shot

of

live

websites

with

their

reconstructions

We

chose

sites

that

were

either

personally

known

to

us

or

randomly

sampled

from

dmoz.org

The

websites

-LRB-

some

were

actually

subsites

-RRB-

were

predominantly

English

covered

a

range

of

topics

and

were

from

a

number

of

top-level

domains

We

chose

small

-LRB-

URIs

-RRB-

medium

-LRB-

URIs

-RRB-

and

large

-LRB-

URIs

-RRB-

websites

and

we

avoided

websites

that

used

robots.txt

and

Flash

exclusively

as

the

main

interface

In

August

we

downloaded

all

websites

by

starting

at

the

base

URL

and

following

all

links

and

references

that

that

were

in

and

beneath

the

starting

directory

with

no

limit

to

the

path

depth

For

simplicity

we

restricted

the

download

to

port

and

did

not

follow

links

to

other

hosts

within

the

same

domain

name

So

if

the

base

URL

for

the

website

was

only

URLs

matching

http

were

downloaded

Warrick

uses

the

same

default

setting

for

reconstructing

websites

Immediately

after

downloading

the

websites

we

reconstructed

five

different

versions

for

each

of

the

websites

four

using

each

web

repository

separately

and

one

using

all

web

repositories

together

The

different

reconstructions

helped

to

show

how

effective

individual

web

repositories

could

reconstruct

a

website

versus

the

aggregate

of

all

four

web

repositories

We

present

of

the

results

of

the

aggregate

reconstructions

in

Table

ordered

by

percent

of

recovered

URIs

The

complete

results

can

be

seen

in

-LSB-

-RSB-

The

PR

column

is

Google

's

PageRank

-LRB-

with

being

the

most

important

-RRB-

for

the

root

page

of

each

website

at

the

time

of

the

experiments

-LRB-

MSN

and

Yahoo

do

not

publicly

disclose

their

importance

metric

-RRB-

For

each

website

the

total

number

of

resources

in

the

website

is

shown

along

with

the

total

number

of

resources

that

were

recovered

and

the

percentage

Resources

are

also

totalled

by

MIME

type

The

difference

vector

for

the

website

accounts

for

recovered

files

that

were

added

The

Almost

identical

column

of

Table

shows

the

percentage

of

text-based

resources

-LRB-

e.g.

HTML

PDF

PostScript

Word

PowerPoint

Excel

-RRB-

that

were

almost

identical

to

the

originals

The

last

column

shows

the

reconstruction

figure

for

each

website

if

these

almost

identical

resources

are

moved

from

the

Changed

category

to

Identical

category

We

considered

two

text-based

resources

to

be

almost

identical

if

they

shared

at

least

of

their

shingles

of

size

Shingling

-LRB-

as

proposed

by

Broder

et

al.

-LSB-

-RSB-

-RRB-

is

a

popular

method

for

quantifying

similarity

of

text

documents

when

word-order

is

important

-LSB-

-RSB-

We

did

not

use

any

image

similarity

metrics

We

were

able

to

recover

more

than

of

the

original

resources

from

a

quarter

of

the

websites

For

three

quarters

of

the

websites

we

recovered

more

than

half

of

the

resources

On

average

we

were

able

to

recover

of

the

website

resources

-LRB-

median

-RRB-

Of

those

resources

recovered

of

them

on

average

were

not

byte-for-byte

identical

A

majority

-LRB-

-RRB-

of

the

changed

text-based

files

were

almost

identical

to

the

originals

-LRB-

having

of

their

shingles

in

common

-RRB-

of

the

websites

had

obtained

additional

files

when

reconstructed

which

accounted

for

of

the

total

number

of

files

reconstructed

per

website

When

all

website

resources

are

aggregated

together

and

examined

dynamic

pages

-LRB-

those

that

contained

a

in

the

URL

-RRB-

were

significantly

less

likely

to

be

recovered

than

resources

that

did

not

have

a

query

string

-LRB-

vs.

-RRB-

URLs

with

a

path

depth

greater

than

three

were

also

less

likely

to

be

recovered

-LRB-

vs.

-RRB-

A

chi-square

analysis

confirms

the

significance

of

these

findings

-LRB-

p

-RRB-

We

were

unable

to

find

any

correlation

between

percentage

of

recovered

resources

with

PageRank

or

website

size

The

success

of

recovering

resources

based

on

their

MIME

type

is

plotted

in

Figure

The

percentage

of

resources

html

images

pdf

other

ms

MIME

type

groups

Nu

m

b

er

o

f

reso

u

r

ces

Ave

of

resources

in

original

websites

Aggregate

recon

IA

recon

Google

recon

MSN

recon

Yahoo!

recon

Figure

Recovery

success

by

MIME

type

that

were

recovered

from

the

five

different

website

reconstructions

we

performed

-LRB-

one

using

all

four

web

repositories

and

four

using

each

web

repository

individually

-RRB-

are

shown

along

with

the

average

number

of

resources

making

up

the

downloaded

-LRB-

or

original

-RRB-

websites

A

majority

-LRB-

-RRB-

of

the

resources

making

up

the

original

websites

are

HTML

and

images

We

were

much

more

successful

at

recovering

HTML

resources

than

images

we

recovered

of

the

HTML

resources

for

of

the

websites

-LRB-

-RRB-

using

all

four

web

repositories

It

is

likely

we

recovered

fewer

images

because

MSN

can

not

be

used

to

recover

images

and

as

our

caching

experiment

revealed

images

are

also

much

less

likely

to

be

cached

than

other

resource

types

Figure

also

emphasizes

the

importance

of

using

all

four

web

repositories

when

reconstructing

a

website

By

just

using

IA

or

just

using

Google

many

resources

will

not

be

recovered

This

is

further

illustrated

by

Figure

which

shows

the

percentage

of

each

web

repository

's

contribution

in

the

aggregate

reconstructions

-LRB-

sites

are

ordered

by

number

of

URIs

-RRB-

Although

Google

was

the

largest

overall

contributor

to

the

website

reconstructions

-LRB-

providing

of

the

resources

-RRB-

they

provided

of

the

resources

for

site

and

provided

less

than

of

the

resources

for

of

the

reconstructions

MSN

contributed

on

average

of

the

resources

IA

was

third

with

and

Yahoo

was

last

with

a

contribution

rate

Yahoo

's

poor

contribution

rate

is

likely

due

to

their

spotty

cache

access

as

exhibited

in

our

caching

experiment

-LRB-

Figure

-RRB-

and

because

last-modified

datestamps

are

frequently

older

than

last-cached

datestamps

-LRB-

Warrick

chooses

resources

with

the

most

recent

datestamps

-RRB-

The

amount

of

time

and

the

number

of

queries

required

to

reconstruct

all

websites

-LRB-

using

all

repositories

-RRB-

is

shown

in

Figure

Here

we

see

almost

a

ratio

of

queries

to

seconds

Although

the

size

of

the

original

websites

gets

larger

along

the

x-axis

the

number

of

files

reconstructed

and

the

number

of

resources

held

in

each

web

repository

determine

how

many

queries

are

performed

In

of

our

reconstructions

did

we

exceed

the

daily

query

limit

of

any

of

the

web

repositories

We

have

made

Warrick

available

on

the

Web

and

it

has

been

used

to

reconstruct

several

websites

have

been

lost

due

to

fire

hard-drive

crashes

death

of

the

website

owner

fmccown/warrick

Reconstructed

w

ebsites

C

ont

r

i

but

i

o

n

Yahoo

IA

MSN

Google

Figure

Web

repositories

contributing

to

each

website

reconstruction

Reconstructed

W

ebsites

N

u

m

b

e

r

of

que

r

i

e

s

Ti

m

e

-LRB-

s

e

c

-RRB-

queries

time

Figure

Number

of

queries

performed

and

time

taken

to

reconstruct

websites

hacking

and

discontinued

charitable

website

hosting

-LSB-

-RSB-

Although

the

reconstructions

have

not

been

complete

individuals

are

very

thankful

to

have

recovered

any

resources

at

all

when

faced

with

total

loss

There

are

numerous

improvements

we

are

making

to

Warrick

including

an

API

for

easier

inclusion

of

new

web

repositories

and

new

methods

for

discovering

more

resources

within

a

web

repository

-LSB-

-RSB-

We

are

planning

on

reconstructing

a

larger

sample

from

the

Web

to

discover

the

website

characteristics

that

allow

for

more

effective

lazy

recovery

Discovering

such

characteristics

will

allow

us

to

create

guidelines

for

webmasters

to

ensure

better

lazy

preservation

of

their

sites

Our

next

experiment

will

take

into

account

rate

of

change

and

reconstruction

differences

over

time

We

are

also

interested

in

recovering

the

server-side

components

-LRB-

CGI

programs

databases

etc.

-RRB-

of

a

lost

website

We

are

investigating

methods

to

inject

server-side

components

into

indexable

content

using

erasure

codes

-LRB-

popular

with

RAID

systems

-LSB-

-RSB-

-RRB-

so

they

can

be

recovered

from

web

repositories

when

only

a

subset

of

pages

can

be

found

A

web-repository

crawler

could

be

used

in

the

future

to

safeguard

websites

that

are

at

risk

of

being

lost

When

a

website

is

detected

as

being

lost

a

reconstruction

could

be

initiated

to

preserve

what

is

left

of

the

site

Additionally

websites

in

countries

that

are

targeted

by

political

censorship

could

be

reconstructed

at

safe

locations

Lazy

preservation

is

a

best-effort

wide-coverage

digital

preservation

service

that

may

be

used

as

a

last

resort

when

website

backups

are

unavailable

It

is

not

a

substitute

for

digital

preservation

infrastructure

and

policy

Web

repositories

may

not

crawl

orphan

pages

protected

pages

-LRB-

e.g.

robots.txt

password

IP

-RRB-

very

large

pages

pages

deep

in

a

web

collection

or

links

influenced

by

JavaScript

Flash

or

session

IDs

If

a

web

repository

will

not

or

can

not

crawl

and

cache

a

resource

it

can

not

be

recovered

We

have

measured

the

ability

of

Google

MSN

and

Yahoo

to

cache

four

synthetic

web

collections

over

a

period

of

four

months

We

measured

web

resources

to

be

vulnerable

for

as

little

as

days

and

in

the

worst

case

as

long

as

our

day

test

period

More

encouragingly

many

HTML

resources

were

recoverable

for

days

on

average

after

being

deleted

from

the

web

server

Google

proved

to

be

the

most

consistent

at

caching

our

synthetic

web

collections

We

have

also

used

our

web-repository

crawler

to

reconstruct

a

variety

of

actual

websites

with

varying

success

HTML

resources

were

the

most

numerous

-LRB-

-RRB-

type

of

resource

in

our

collection

of

websites

and

were

the

most

successfully

recoverable

resource

type

-LRB-

recoverable

-RRB-

Images

were

the

second

most

numerous

-LRB-

-RRB-

resource

type

but

they

were

less

successfully

recovered

-LRB-

-RRB-

Dynamic

pages

and

resources

with

path

depths

greater

than

three

were

less

likely

to

be

recovered

Google

was

the

most

frequent

source

for

the

reconstructions

-LRB-

-RRB-

but

MSN

was

a

close

second

-LRB-

-RRB-

followed

by

IA

-LRB-

-RRB-

and

Yahoo

-LRB-

-RRB-

The

probability

of

reconstruction

success

was

not

correlated

with

Google

's

PageRank

or

the

size

of

the

website

-LSB-

-RSB-

H.

Berghel

Responsible

web

caching

Communications

of

the

ACM

-LRB-

-RRB-

-LSB-

-RSB-

K.

Bharat

and

A.

Broder

Mirror

mirror

on

the

web

a

study

of

host

pairs

with

replicated

content

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

A.

Z.

Broder

S.

C.

Glassman

M.

S.

Manasse

and

G.

Zweig

Syntactic

clustering

of

the

Web

Computer

Networks

ISDN

Systems

-LRB-

-RRB-

-LSB-

-RSB-

M.

Burner

Crawling

towards

eternity

Building

an

archive

of

the

world

wide

web

Web

Techniques

Magazine

-LRB-

-RRB-

-LSB-

-RSB-

F.

Can

R.

Nuray

and

A.

B.

Sevdik

Automatic

performance

evaluation

of

web

search

engines

Info

Processing

Management

-LRB-

-RRB-

-LSB-

-RSB-

J.

Cho

N.

Shivakumar

and

H.

Garcia-Molina

Finding

replicated

web

collections

In

Proceedings

of

SIGMOD

pages

-LSB-

-RSB-

B.

F.

Cooper

and

H.

Garcia-Molina

Infomonitor

Unobtrusively

archiving

a

World

Wide

Web

server

International

Journal

on

Digital

Libraries

-LRB-

-RRB-

April

-LSB-

-RSB-

M.

Day

Collecting

and

preserving

the

World

Wide

Web

http

-LSB-

-RSB-

C.

E.

Dyreson

H.

Lin

and

Y.

Wang

Managing

versions

of

web

documents

in

a

transaction-time

web

server

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

D.

Fetterly

M.

Manasse

and

M.

Najork

Spam

damn

spam

and

statistics

using

statistical

analysis

to

locate

spam

web

pages

In

Proceedings

of

WebDB

pages

-LSB-

-RSB-

D.

Fetterly

M.

Manasse

M.

Najork

and

J.

Wiener

A

large-scale

study

of

the

evolution

of

web

pages

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

Google

Sitemap

Protocol

-LSB-

-RSB-

Google

webmaster

help

center

Webmaster

guidelines

bin/answer.py

answer

-LSB-

-RSB-

M.

Gordon

and

P.

Pathak

Finding

information

on

the

World

Wide

Web

the

retrieval

effectiveness

of

search

engines

Inf

Process

Manage.

-LRB-

-RRB-

-LSB-

-RSB-

A.

Gulli

and

A.

Signorini

The

indexable

web

is

more

than

billion

pages

In

Proceedings

of

WWW

pages

May

-LSB-

-RSB-

Internet

Archive

FAQ

How

can

I

get

my

site

included

in

the

Archive

-LSB-

-RSB-

D.

Lewandowski

H.

Wahlig

and

G.

Meyer-Beautor

The

freshness

of

Web

search

engine

databases

Journal

of

Information

Science

-LRB-

-RRB-

Apr

-LSB-

-RSB-

F.

McCown

X.

Liu

M.

L.

Nelson

and

M.

Zubair

Search

engine

coverage

of

the

OAI-PMH

corpus

IEEE

Internet

Computing

-LRB-

-RRB-

Mar/Apr

-LSB-

-RSB-

F.

McCown

and

M.

L.

Nelson

Evaluation

of

crawling

policies

for

a

web-repository

crawler

In

Proceedings

of

HYPERTEXT

pages

-LSB-

-RSB-

F.

McCown

J.

A.

Smith

M.

L.

Nelson

and

J.

Bollen

Reconstructing

websites

for

the

lazy

webmaster

Technical

report

Old

Dominion

University

-LSB-

-RSB-

A.

Ntoulas

J.

Cho

and

C.

Olston

What

's

new

on

the

Web

The

evolution

of

the

Web

from

a

search

engine

perspective

In

Proceedings

of

WWW

pages

-LSB-

-RSB-

J.

S.

Plank

A

tutorial

on

Reed-Solomon

coding

for

fault-tolerance

in

RAID-like

systems

Software

Practice

and

Experience

-LRB-

-RRB-

-LSB-

-RSB-

H.

C.

Rao

Y.

Chen

and

M.

Chen

A

proxy-based

personal

web

archiving

service

SIGOPS

Operating

Systems

Review

-LRB-

-RRB-

-LSB-

-RSB-

V.

Reich

and

D.

S.

Rosenthal

LOCKSS

A

permanent

web

publishing

and

access

system

D-Lib

Magazine

-LRB-

-RRB-

-LSB-

-RSB-

A.

Ross

Internet

Archive

forums

Web

forum

posting

Oct

post-view.php

id

-LSB-

-RSB-

J.

A.

Smith

F.

McCown

and

M.

L.

Nelson

Observed

web

robot

behavior

on

decaying

web

subsites

D-Lib

Magazine

-LRB-

-RRB-

Feb

-LSB-

-RSB-

M.

Weideman

and

M.

Mgidana

Website

navigation

architectures

and

their

effect

on

website

visibility

a

literature

survey

In

Proceedings

of

SAICSIT

pages

-LSB-

-RSB-

J.

Zhang

and

A.

Dimitroff

The

impact

of

webpage

content

characteristics

on

webpage

visibility

in

search

engine

results

-LRB-

part

I

-RRB-

Information

Processing

Management

-LRB-

-RRB-

B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O

caching
crawling
search engine
cached resources
lazy preservation
digital preservation
reconstruction
recovery
web repositories