Downloading

Textual

Hidden

Web

Content

Through

Keyword

Queries

An

ever-increasing

amount

of

information

on

the

Web

today

is

available

only

through

search

interfaces

the

users

have

to

type

in

a

set

of

keywords

in

a

search

form

in

order

to

access

the

pages

from

certain

Web

sites

These

pages

are

often

referred

to

as

the

Hidden

Web

or

the

Deep

Web

Since

there

are

no

static

links

to

the

Hidden

Web

pages

search

engines

can

not

discover

and

index

such

pages

and

thus

do

not

return

them

in

the

results

However

according

to

recent

studies

the

content

provided

by

many

Hidden

Web

sites

is

often

of

very

high

quality

and

can

be

extremely

valuable

to

many

users

In

this

paper

we

study

how

we

can

build

an

effective

Hidden

Web

crawler

that

can

autonomously

discover

and

download

pages

from

the

Hidden

Web

Since

the

only

entry

point

to

a

Hidden

Web

site

is

a

query

interface

the

main

challenge

that

a

Hidden

Web

crawler

has

to

face

is

how

to

automatically

generate

meaningful

queries

to

issue

to

the

site

Here

we

provide

a

theoretical

framework

to

investigate

the

query

generation

problem

for

the

Hidden

Web

and

we

propose

effective

policies

for

generating

queries

automatically

Our

policies

proceed

iteratively

issuing

a

different

query

in

every

iteration

We

experimentally

evaluate

the

effectiveness

of

these

policies

on

real

Hidden

Web

sites

and

our

results

are

very

promising

For

instance

in

one

experiment

one

of

our

policies

downloaded

more

than

of

a

Hidden

Web

site

-LRB-

that

contains

million

documents

-RRB-

after

issuing

fewer

than

queries

H.

-LSB-

Information

Systems

-RSB-

Digital

Libraries

H.

-LSB-

Information

Systems

-RSB-

Content

Analysis

and

Indexing

H.

-LSB-

Information

Systems

-RSB-

Information

Search

and

Retrieval

Algorithms

Performance

Design

Recent

studies

show

that

a

significant

fraction

of

Web

content

can

not

be

reached

by

following

links

-LSB-

-RSB-

In

particular

a

large

part

of

the

Web

is

hidden

behind

search

forms

and

is

reachable

only

when

users

type

in

a

set

of

keywords

or

queries

to

the

forms

These

pages

are

often

referred

to

as

the

Hidden

Web

-LSB-

-RSB-

or

the

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

JCDL

June

Denver

Colorado

USA

Copyright

ACM

Deep

Web

-LSB-

-RSB-

because

search

engines

typically

can

not

index

the

pages

and

do

not

return

them

in

their

results

-LRB-

thus

the

pages

are

essentially

hidden

from

a

typical

Web

user

-RRB-

According

to

many

studies

the

size

of

the

Hidden

Web

increases

rapidly

as

more

organizations

put

their

valuable

content

online

through

an

easy-to-use

Web

interface

-LSB-

-RSB-

In

-LSB-

-RSB-

Chang

et

al.

estimate

that

well

over

Hidden-Web

sites

currently

exist

on

the

Web

Moreover

the

content

provided

by

many

Hidden-Web

sites

is

often

of

very

high

quality

and

can

be

extremely

valuable

to

many

users

-LSB-

-RSB-

For

example

PubMed

hosts

many

high-quality

papers

on

medical

research

that

were

selected

from

careful

peer-review

processes

while

the

site

of

the

US

Patent

and

Trademarks

Office

makes

existing

patent

documents

available

helping

potential

inventors

examine

prior

art

In

this

paper

we

study

how

we

can

build

a

Hidden-Web

crawler

that

can

automatically

download

pages

from

the

Hidden

Web

so

that

search

engines

can

index

them

Conventional

crawlers

rely

on

the

hyperlinks

on

the

Web

to

discover

pages

so

current

search

engines

can

not

index

the

Hidden-Web

pages

-LRB-

due

to

the

lack

of

links

-RRB-

We

believe

that

an

effective

Hidden-Web

crawler

can

have

a

tremendous

impact

on

how

users

search

information

on

the

Web

Tapping

into

unexplored

information

The

Hidden-Web

crawler

will

allow

an

average

Web

user

to

easily

explore

the

vast

amount

of

information

that

is

mostly

hidden

at

present

Since

a

majority

of

Web

users

rely

on

search

engines

to

discover

pages

when

pages

are

not

indexed

by

search

engines

they

are

unlikely

to

be

viewed

by

many

Web

users

Unless

users

go

directly

to

Hidden-Web

sites

and

issue

queries

there

they

can

not

access

the

pages

at

the

sites

Improving

user

experience

Even

if

a

user

is

aware

of

a

number

of

Hidden-Web

sites

the

user

still

has

to

waste

a

significant

amount

of

time

and

effort

visiting

all

of

the

potentially

relevant

sites

querying

each

of

them

and

exploring

the

result

By

making

the

Hidden-Web

pages

searchable

at

a

central

location

we

can

significantly

reduce

the

user

's

wasted

time

and

effort

in

searching

the

Hidden

Web

Reducing

potential

bias

Due

to

the

heavy

reliance

of

many

Web

users

on

search

engines

for

locating

information

search

engines

influence

how

the

users

perceive

the

Web

-LSB-

-RSB-

Users

do

not

necessarily

perceive

what

actually

exists

on

the

Web

but

what

is

indexed

by

search

engines

-LSB-

-RSB-

According

to

a

recent

article

-LSB-

-RSB-

several

organizations

have

recognized

the

importance

of

bringing

information

of

their

Hidden

Web

sites

onto

the

surface

and

committed

considerable

resources

towards

this

effort

Our

US

Patent

Office

Crawlers

are

the

programs

that

traverse

the

Web

automatically

and

download

pages

for

search

engines

Figure

A

single-attribute

search

interface

Hidden-Web

crawler

attempts

to

automate

this

process

for

Hidden

Web

sites

with

textual

content

thus

minimizing

the

associated

costs

and

effort

required

Given

that

the

only

entry

to

Hidden

Web

pages

is

through

querying

a

search

form

there

are

two

core

challenges

to

implementing

an

effective

Hidden

Web

crawler

-LRB-

a

-RRB-

The

crawler

has

to

be

able

to

understand

and

model

a

query

interface

and

-LRB-

b

-RRB-

The

crawler

has

to

come

up

with

meaningful

queries

to

issue

to

the

query

interface

The

first

challenge

was

addressed

by

Raghavan

and

Garcia-Molina

in

-LSB-

-RSB-

where

a

method

for

learning

search

interfaces

was

presented

Here

we

present

a

solution

to

the

second

challenge

i.e.

how

a

crawler

can

automatically

generate

queries

so

that

it

can

discover

and

download

the

Hidden

Web

pages

Clearly

when

the

search

forms

list

all

possible

values

for

a

query

-LRB-

e.g.

through

a

drop-down

list

-RRB-

the

solution

is

straightforward

We

exhaustively

issue

all

possible

queries

one

query

at

a

time

When

the

query

forms

have

a

free

text

input

however

an

infinite

number

of

queries

are

possible

so

we

can

not

exhaustively

issue

all

possible

queries

In

this

case

what

queries

should

we

pick

Can

the

crawler

automatically

come

up

with

meaningful

queries

without

understanding

the

semantics

of

the

search

form

In

this

paper

we

provide

a

theoretical

framework

to

investigate

the

Hidden-Web

crawling

problem

and

propose

effective

ways

of

generating

queries

automatically

We

also

evaluate

our

proposed

solutions

through

experiments

conducted

on

real

Hidden-Web

sites

In

summary

this

paper

makes

the

following

contributions

We

present

a

formal

framework

to

study

the

problem

of

Hidden-Web

crawling

-LRB-

Section

-RRB-

We

investigate

a

number

of

crawling

policies

for

the

Hidden

Web

including

the

optimal

policy

that

can

potentially

download

the

maximum

number

of

pages

through

the

minimum

number

of

interactions

Unfortunately

we

show

that

the

optimal

policy

is

NP-hard

and

can

not

be

implemented

in

practice

-LRB-

Section

-RRB-

We

propose

a

new

adaptive

policy

that

approximates

the

optimal

policy

Our

adaptive

policy

examines

the

pages

returned

from

previous

queries

and

adapts

its

query-selection

policy

automatically

based

on

them

-LRB-

Section

-RRB-

We

evaluate

various

crawling

policies

through

experiments

on

real

Web

sites

Our

experiments

will

show

the

relative

advantages

of

various

crawling

policies

and

demonstrate

their

potential

The

results

from

our

experiments

are

very

promising

In

one

experiment

for

example

our

adaptive

policy

downloaded

more

than

of

the

pages

within

PubMed

-LRB-

that

contains

million

documents

-RRB-

after

it

issued

fewer

than

queries

In

this

section

we

present

a

formal

framework

for

the

study

of

the

Hidden-Web

crawling

problem

In

Section

we

describe

our

assumptions

on

Hidden-Web

sites

and

explain

how

users

interact

with

the

sites

Based

on

this

interaction

model

we

present

a

high-level

algorithm

for

a

Hidden-Web

crawler

in

Section

Finally

in

Section

we

formalize

the

Hidden-Web

crawling

problem

Hidden-Web

database

model

There

exists

a

variety

of

Hidden

Web

sources

that

provide

information

on

a

multitude

of

topics

Depending

on

the

type

of

information

we

may

categorize

a

Hidden-Web

site

either

as

a

textual

database

or

a

structured

database

A

textual

database

is

a

site

that

Figure

A

multi-attribute

search

interface

mainly

contains

plain-text

documents

such

as

PubMed

and

Lexis-Nexis

-LRB-

an

online

database

of

legal

documents

-LSB-

-RSB-

-RRB-

Since

plain-text

documents

do

not

usually

have

well-defined

structure

most

textual

databases

provide

a

simple

search

interface

where

users

type

a

list

of

keywords

in

a

single

search

box

-LRB-

Figure

-RRB-

In

contrast

a

structured

database

often

contains

multi-attribute

relational

data

-LRB-

e.g.

a

book

on

the

Amazon

Web

site

may

have

the

fields

title

Harry

Potter

author

J.K.

Rowling

and

isbn

-RRB-

and

supports

multi-attribute

search

interfaces

-LRB-

Figure

-RRB-

In

this

paper

we

will

mainly

focus

on

textual

databases

that

support

single-attribute

keyword

queries

We

discuss

how

we

can

extend

our

ideas

for

the

textual

databases

to

multi-attribute

structured

databases

in

Section

Typically

the

users

need

to

take

the

following

steps

in

order

to

access

pages

in

a

Hidden-Web

database

Step

First

the

user

issues

a

query

say

liver

through

the

search

interface

provided

by

the

Web

site

-LRB-

such

as

the

one

shown

in

Figure

-RRB-

Step

Shortly

after

the

user

issues

the

query

she

is

presented

with

a

result

index

page

That

is

the

Web

site

returns

a

list

of

links

to

potentially

relevant

Web

pages

as

shown

in

Figure

-LRB-

a

-RRB-

Step

From

the

list

in

the

result

index

page

the

user

identifies

the

pages

that

look

interesting

and

follows

the

links

Clicking

on

a

link

leads

the

user

to

the

actual

Web

page

such

as

the

one

shown

in

Figure

-LRB-

b

-RRB-

that

the

user

wants

to

look

at

A

generic

Hidden

Web

crawling

algorithm

Given

that

the

only

entry

to

the

pages

in

a

Hidden-Web

site

is

its

search

from

a

Hidden-Web

crawler

should

follow

the

three

steps

described

in

the

previous

section

That

is

the

crawler

has

to

generate

a

query

issue

it

to

the

Web

site

download

the

result

index

page

and

follow

the

links

to

download

the

actual

pages

In

most

cases

a

crawler

has

limited

time

and

network

resources

so

the

crawler

repeats

these

steps

until

it

uses

up

its

resources

In

Figure

we

show

the

generic

algorithm

for

a

Hidden-Web

crawler

For

simplicity

we

assume

that

the

Hidden-Web

crawler

issues

single-term

queries

only

The

crawler

first

decides

which

query

term

it

is

going

to

use

-LRB-

Step

-LRB-

-RRB-

-RRB-

issues

the

query

and

retrieves

the

result

index

page

-LRB-

Step

-LRB-

-RRB-

-RRB-

Finally

based

on

the

links

found

on

the

result

index

page

it

downloads

the

Hidden

Web

pages

from

the

site

-LRB-

Step

-LRB-

-RRB-

-RRB-

This

same

process

is

repeated

until

all

the

available

resources

are

used

up

-LRB-

Step

-LRB-

-RRB-

-RRB-

Given

this

algorithm

we

can

see

that

the

most

critical

decision

that

a

crawler

has

to

make

is

what

query

to

issue

next

If

the

crawler

can

issue

successful

queries

that

will

return

many

matching

pages

the

crawler

can

finish

its

crawling

early

on

using

minimum

resources

In

contrast

if

the

crawler

issues

completely

irrelevant

queries

that

do

not

return

any

matching

pages

it

may

waste

all

of

its

resources

simply

issuing

queries

without

ever

retrieving

actual

pages

Therefore

how

the

crawler

selects

the

next

query

can

greatly

affect

its

effectiveness

In

the

next

section

we

formalize

this

query

selection

problem

For

most

Web

sites

that

assume

AND

for

multi-keyword

queries

single-term

queries

return

the

maximum

number

of

results

Extending

our

work

to

multi-keyword

queries

is

straightforward

-LRB-

a

-RRB-

List

of

matching

pages

for

query

liver

-LRB-

b

-RRB-

The

first

matching

page

for

liver

Figure

Pages

from

the

PubMed

Web

site

A

LGORITHM

Crawling

a

Hidden

Web

site

Procedure

-LRB-

-RRB-

while

-LRB-

there

are

available

resources

-RRB-

do

select

a

term

to

send

to

the

site

-LRB-

-RRB-

q

i

SelectTerm

-LRB-

-RRB-

send

query

and

acquire

result

index

page

-LRB-

-RRB-

R

-LRB-

q

i

-RRB-

QueryWebSite

-LRB-

q

i

-RRB-

download

the

pages

of

interest

-LRB-

-RRB-

Download

-LRB-

R

-LRB-

q

i

-RRB-

-RRB-

-LRB-

-RRB-

done

Figure

Algorithm

for

crawling

a

Hidden

Web

site

S

q

q

q

q

Figure

A

set-formalization

of

the

optimal

query

selection

problem

Problem

formalization

Theoretically

the

problem

of

query

selection

can

be

formalized

as

follows

We

assume

that

the

crawler

downloads

pages

from

a

Web

site

that

has

a

set

of

pages

S

-LRB-

the

rectangle

in

Figure

-RRB-

We

represent

each

Web

page

in

S

as

a

point

-LRB-

dots

in

Figure

-RRB-

Every

potential

query

q

i

that

we

may

issue

can

be

viewed

as

a

subset

of

S

containing

all

the

points

-LRB-

pages

-RRB-

that

are

returned

when

we

issue

q

i

to

the

site

Each

subset

is

associated

with

a

weight

that

represents

the

cost

of

issuing

the

query

Under

this

formalization

our

goal

is

to

find

which

subsets

-LRB-

queries

-RRB-

cover

the

maximum

number

of

points

-LRB-

Web

pages

-RRB-

with

the

minimum

total

weight

-LRB-

cost

-RRB-

This

problem

is

equivalent

to

the

set-covering

problem

in

graph

theory

-LSB-

-RSB-

There

are

two

main

difficulties

that

we

need

to

address

in

this

formalization

First

in

a

practical

situation

the

crawler

does

not

know

which

Web

pages

will

be

returned

by

which

queries

so

the

subsets

of

S

are

not

known

in

advance

Without

knowing

these

subsets

the

crawler

can

not

decide

which

queries

to

pick

to

maximize

the

coverage

Second

the

set-covering

problem

is

known

to

be

NP-Hard

-LSB-

-RSB-

so

an

efficient

algorithm

to

solve

this

problem

optimally

in

polynomial

time

has

yet

to

be

found

In

this

paper

we

will

present

an

approximation

algorithm

that

can

find

a

near-optimal

solution

at

a

reasonable

computational

cost

Our

algorithm

leverages

the

observation

that

although

we

do

not

know

which

pages

will

be

returned

by

each

query

q

i

that

we

issue

we

can

predict

how

many

pages

will

be

returned

Based

on

this

information

our

query

selection

algorithm

can

then

select

the

best

queries

that

cover

the

content

of

the

Web

site

We

present

our

prediction

method

and

our

query

selection

algorithm

in

Section

Performance

Metric

Before

we

present

our

ideas

for

the

query

selection

problem

we

briefly

discuss

some

of

our

notation

and

the

cost/performance

metrics

Given

a

query

q

i

we

use

P

-LRB-

q

i

-RRB-

to

denote

the

fraction

of

pages

that

we

will

get

back

if

we

issue

query

q

i

to

the

site

For

example

if

a

Web

site

has

pages

in

total

and

if

pages

are

returned

for

the

query

q

i

medicine

then

P

-LRB-

q

i

-RRB-

We

use

P

-LRB-

q

q

-RRB-

to

represent

the

fraction

of

pages

that

are

returned

from

both

q

and

q

-LRB-

i.e.

the

intersection

of

P

-LRB-

q

-RRB-

and

P

-LRB-

q

-RRB-

-RRB-

Similarly

we

use

P

-LRB-

q

q

-RRB-

to

represent

the

fraction

of

pages

that

are

returned

from

either

q

or

q

-LRB-

i.e.

the

union

of

P

-LRB-

q

-RRB-

and

P

-LRB-

q

-RRB-

-RRB-

We

also

use

Cost

-LRB-

q

i

-RRB-

to

represent

the

cost

of

issuing

the

query

q

i

Depending

on

the

scenario

the

cost

can

be

measured

either

in

time

network

bandwidth

the

number

of

interactions

with

the

site

or

it

can

be

a

function

of

all

of

these

As

we

will

see

later

our

proposed

algorithms

are

independent

of

the

exact

cost

function

In

the

most

common

case

the

query

cost

consists

of

a

number

of

factors

including

the

cost

for

submitting

the

query

to

the

site

retrieving

the

result

index

page

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

and

downloading

the

actual

pages

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

We

assume

that

submitting

a

query

incurs

a

fixed

cost

of

c

q

The

cost

for

downloading

the

result

index

page

is

proportional

to

the

number

of

matching

documents

to

the

query

while

the

cost

c

d

for

downloading

a

matching

document

is

also

fixed

Then

the

overall

cost

of

query

q

i

is

Cost

-LRB-

q

i

-RRB-

c

q

c

r

P

-LRB-

q

i

-RRB-

c

d

P

-LRB-

q

i

-RRB-

-LRB-

-RRB-

In

certain

cases

some

of

the

documents

from

q

i

may

have

already

been

downloaded

from

previous

queries

In

this

case

the

crawler

may

skip

downloading

these

documents

and

the

cost

of

q

i

can

be

Cost

-LRB-

q

i

-RRB-

c

q

c

r

P

-LRB-

q

i

-RRB-

c

d

P

new

-LRB-

q

i

-RRB-

-LRB-

-RRB-

Here

we

use

P

new

-LRB-

q

i

-RRB-

to

represent

the

fraction

of

the

new

documents

from

q

i

that

have

not

been

retrieved

from

previous

queries

Later

in

Section

we

will

study

how

we

can

estimate

P

-LRB-

q

i

-RRB-

and

P

new

-LRB-

q

i

-RRB-

to

estimate

the

cost

of

q

i

Since

our

algorithms

are

independent

of

the

exact

cost

function

we

will

assume

a

generic

cost

function

Cost

-LRB-

q

i

-RRB-

in

this

paper

When

we

need

a

concrete

cost

function

however

we

will

use

Equation

Given

the

notation

we

can

formalize

the

goal

of

a

Hidden-Web

crawler

as

follows

P

ROBLEM

Find

the

set

of

queries

q

q

n

that

maximizes

P

-LRB-

q

q

n

-RRB-

under

the

constraint

n

i

Cost

-LRB-

q

i

-RRB-

t.

Here

t

is

the

maximum

download

resource

that

the

crawler

has

How

should

a

crawler

select

the

queries

to

issue

Given

that

the

goal

is

to

download

the

maximum

number

of

unique

documents

from

a

textual

database

we

may

consider

one

of

the

following

options

Random

We

select

random

keywords

from

say

an

English

dictionary

and

issue

them

to

the

database

The

hope

is

that

a

random

query

will

return

a

reasonable

number

of

matching

documents

Generic-frequency

We

analyze

a

generic

document

corpus

collected

elsewhere

-LRB-

say

from

the

Web

-RRB-

and

obtain

the

generic

frequency

distribution

of

each

keyword

Based

on

this

generic

distribution

we

start

with

the

most

frequent

keyword

issue

it

to

the

Hidden-Web

database

and

retrieve

the

result

We

then

continue

to

the

second-most

frequent

keyword

and

repeat

this

process

until

we

exhaust

all

download

resources

The

hope

is

that

the

frequent

keywords

in

a

generic

corpus

will

also

be

frequent

in

the

Hidden-Web

database

returning

many

matching

documents

Adaptive

We

analyze

the

documents

returned

from

the

previous

queries

issued

to

the

Hidden-Web

database

and

estimate

which

keyword

is

most

likely

to

return

the

most

documents

Based

on

this

analysis

we

issue

the

most

promising

query

and

repeat

the

process

Among

these

three

general

policies

we

may

consider

the

random

policy

as

the

base

comparison

point

since

it

is

expected

to

perform

the

worst

Between

the

generic-frequency

and

the

adaptive

policies

both

policies

may

show

similar

performance

if

the

crawled

database

has

a

generic

document

collection

without

a

specialized

topic

The

adaptive

policy

however

may

perform

significantly

better

than

the

generic-frequency

policy

if

the

database

has

a

very

specialized

collection

that

is

different

from

the

generic

corpus

We

will

experimentally

compare

these

three

policies

in

Section

While

the

first

two

policies

-LRB-

random

and

generic-frequency

policies

-RRB-

are

easy

to

implement

we

need

to

understand

how

we

can

analyze

the

downloaded

pages

to

identify

the

most

promising

query

in

order

to

implement

the

adaptive

policy

We

address

this

issue

in

the

rest

of

this

section

Estimating

the

number

of

matching

pages

In

order

to

identify

the

most

promising

query

we

need

to

estimate

how

many

new

documents

we

will

download

if

we

issue

the

query

q

i

as

the

next

query

That

is

assuming

that

we

have

issued

queries

q

q

we

need

to

estimate

P

-LRB-

q

q

q

i

-RRB-

for

every

potential

next

query

q

i

and

compare

this

value

In

estimating

this

number

we

note

that

we

can

rewrite

P

-LRB-

q

q

q

i

-RRB-

as

P

-LRB-

-LRB-

q

q

-RRB-

q

i

-RRB-

P

-LRB-

q

q

-RRB-

P

-LRB-

q

i

-RRB-

P

-LRB-

-LRB-

q

q

-RRB-

q

i

-RRB-

P

-LRB-

q

q

-RRB-

P

-LRB-

q

i

-RRB-

P

-LRB-

q

q

-RRB-

P

-LRB-

q

i

q

q

-RRB-

-LRB-

-RRB-

In

the

above

formula

note

that

we

can

precisely

measure

P

-LRB-

q

q

-RRB-

and

P

-LRB-

q

i

q

q

-RRB-

by

analyzing

previously-downloaded

pages

We

know

P

-LRB-

q

q

-RRB-

the

union

of

all

pages

downloaded

from

q

q

since

we

have

already

issued

q

q

and

downloaded

the

matching

pages

We

can

also

measure

P

-LRB-

q

i

q

q

-RRB-

the

probability

that

q

i

appears

in

the

pages

from

q

q

by

counting

how

many

times

q

i

appears

in

the

pages

from

q

q

Therefore

we

only

need

to

estimate

P

-LRB-

q

i

-RRB-

to

evaluate

P

-LRB-

q

q

i

-RRB-

We

may

consider

a

number

of

different

ways

to

estimate

P

-LRB-

q

i

-RRB-

including

the

following

Independence

estimator

We

assume

that

the

appearance

of

the

term

q

i

is

independent

of

the

terms

q

q

That

is

we

assume

that

P

-LRB-

q

i

-RRB-

P

-LRB-

q

i

q

q

-RRB-

Zipf

estimator

In

-LSB-

-RSB-

Ipeirotis

et

al.

proposed

a

method

to

estimate

how

many

times

a

particular

term

occurs

in

the

entire

corpus

based

on

a

subset

of

documents

from

the

corpus

Their

method

exploits

the

fact

that

the

frequency

of

terms

inside

text

collections

follows

a

power

law

distribution

-LSB-

-RSB-

That

is

if

we

rank

all

terms

based

on

their

occurrence

frequency

-LRB-

with

the

most

frequent

term

having

a

rank

of

second

most

frequent

a

rank

of

etc.

-RRB-

then

the

frequency

f

of

a

term

inside

the

text

collection

is

given

by

f

-LRB-

r

-RRB-

-LRB-

-RRB-

where

r

is

the

rank

of

the

term

and

and

are

constants

that

depend

on

the

text

collection

Their

main

idea

is

-LRB-

-RRB-

to

estimate

the

three

parameters

and

based

on

the

subset

of

documents

that

we

have

downloaded

from

previous

queries

and

-LRB-

-RRB-

use

the

estimated

parameters

to

predict

f

given

the

ranking

r

of

a

term

within

the

subset

For

a

more

detailed

description

on

how

we

can

use

this

method

to

estimate

P

-LRB-

q

i

-RRB-

we

refer

the

reader

to

the

extended

version

of

this

paper

-LSB-

-RSB-

After

we

estimate

P

-LRB-

q

i

-RRB-

and

P

-LRB-

q

i

q

q

-RRB-

values

we

can

calculate

P

-LRB-

q

q

i

-RRB-

In

Section

we

explain

how

we

can

efficiently

compute

P

-LRB-

q

i

q

q

-RRB-

by

maintaining

a

succinct

summary

table

In

the

next

section

we

first

examine

how

we

can

use

this

value

to

decide

which

query

we

should

issue

next

to

the

Hidden

Web

site

Query

selection

algorithm

The

goal

of

the

Hidden-Web

crawler

is

to

download

the

maximum

number

of

unique

documents

from

a

database

using

its

limited

download

resources

Given

this

goal

the

Hidden-Web

crawler

has

to

take

two

factors

into

account

-LRB-

-RRB-

the

number

of

new

documents

that

can

be

obtained

from

the

query

q

i

and

-LRB-

-RRB-

the

cost

of

issuing

the

query

q

i

For

example

if

two

queries

q

i

and

q

j

incur

the

same

cost

but

q

i

returns

more

new

pages

than

q

j

q

i

is

more

desirable

than

q

j

Similarly

if

q

i

and

q

j

return

the

same

number

of

new

documents

but

q

i

incurs

less

cost

then

q

j

q

i

is

more

desirable

Based

on

this

observation

the

Hidden-Web

crawler

may

use

the

following

efficiency

metric

to

quantify

the

desirability

of

the

query

q

i

Efficiency

-LRB-

q

i

-RRB-

P

new

-LRB-

q

i

-RRB-

Cost

-LRB-

q

i

-RRB-

Here

P

new

-LRB-

q

i

-RRB-

represents

the

amount

of

new

documents

returned

for

q

i

-LRB-

the

pages

that

have

not

been

returned

for

previous

queries

-RRB-

Cost

-LRB-

q

i

-RRB-

represents

the

cost

of

issuing

the

query

q

i

Intuitively

the

efficiency

of

q

i

measures

how

many

new

documents

are

retrieved

per

unit

cost

and

can

be

used

as

an

indicator

of

For

exact

estimation

we

need

to

know

the

total

number

of

pages

in

the

site

However

in

order

to

compare

only

relative

values

among

queries

this

information

is

not

actually

needed

A

LGORITHM

Greedy

SelectTerm

-LRB-

-RRB-

Parameters

T

The

list

of

potential

query

keywords

Procedure

-LRB-

-RRB-

Foreach

t

k

in

T

do

-LRB-

-RRB-

Estimate

Efficiency

-LRB-

t

k

-RRB-

P

new

-LRB-

t

k

-RRB-

Cost

-LRB-

t

k

-RRB-

-LRB-

-RRB-

done

-LRB-

-RRB-

return

t

k

with

maximum

Efficiency

-LRB-

t

k

-RRB-

Figure

Algorithm

for

selecting

the

next

query

term

how

well

our

resources

are

spent

when

issuing

q

i

Thus

the

Hidden

Web

crawler

can

estimate

the

efficiency

of

every

candidate

q

i

and

select

the

one

with

the

highest

value

By

using

its

resources

more

efficiently

the

crawler

may

eventually

download

the

maximum

number

of

unique

documents

In

Figure

we

show

the

query

selection

function

that

uses

the

concept

of

efficiency

In

principle

this

algorithm

takes

a

greedy

approach

and

tries

to

maximize

the

potential

gain

in

every

step

We

can

estimate

the

efficiency

of

every

query

using

the

estimation

method

described

in

Section

That

is

the

size

of

the

new

documents

from

the

query

q

i

P

new

-LRB-

q

i

-RRB-

is

P

new

-LRB-

q

i

-RRB-

P

-LRB-

q

q

q

i

-RRB-

P

-LRB-

q

q

-RRB-

P

-LRB-

q

i

-RRB-

P

-LRB-

q

q

-RRB-

P

-LRB-

q

i

q

q

-RRB-

from

Equation

where

P

-LRB-

q

i

-RRB-

can

be

estimated

using

one

of

the

methods

described

in

section

We

can

also

estimate

Cost

-LRB-

q

i

-RRB-

sim-ilarly

For

example

if

Cost

-LRB-

q

i

-RRB-

is

Cost

-LRB-

q

i

-RRB-

c

q

c

r

P

-LRB-

q

i

-RRB-

c

d

P

new

-LRB-

q

i

-RRB-

-LRB-

Equation

-RRB-

we

can

estimate

Cost

-LRB-

q

i

-RRB-

by

estimating

P

-LRB-

q

i

-RRB-

and

P

new

-LRB-

q

i

-RRB-

Efficient

calculation

of

query

statistics

In

estimating

the

efficiency

of

queries

we

found

that

we

need

to

measure

P

-LRB-

q

i

q

q

-RRB-

for

every

potential

query

q

i

This

calculation

can

be

very

time-consuming

if

we

repeat

it

from

scratch

for

every

query

q

i

in

every

iteration

of

our

algorithm

In

this

section

we

explain

how

we

can

compute

P

-LRB-

q

i

q

q

-RRB-

efficiently

by

maintaining

a

small

table

that

we

call

a

query

statistics

table

The

main

idea

for

the

query

statistics

table

is

that

P

-LRB-

q

i

q

q

-RRB-

can

be

measured

by

counting

how

many

times

the

keyword

q

i

appears

within

the

documents

downloaded

from

q

q

We

record

these

counts

in

a

table

as

shown

in

Figure

-LRB-

a

-RRB-

The

left

column

of

the

table

contains

all

potential

query

terms

and

the

right

column

contains

the

number

of

previously-downloaded

documents

containing

the

respective

term

For

example

the

table

in

Figure

-LRB-

a

-RRB-

shows

that

we

have

downloaded

documents

so

far

and

the

term

model

appears

in

of

these

documents

Given

this

number

we

can

compute

that

P

-LRB-

model

q

q

-RRB-

We

note

that

the

query

statistics

table

needs

to

be

updated

whenever

we

issue

a

new

query

q

i

and

download

more

documents

This

update

can

be

done

efficiently

as

we

illustrate

in

the

following

example

E

XAMPLE

After

examining

the

query

statistics

table

of

Figure

-LRB-

a

-RRB-

we

have

decided

to

use

the

term

computer

as

our

next

query

q

i

From

the

new

query

q

i

computer

we

downloaded

more

new

pages

Out

of

these

contain

the

keyword

model

Term

t

k

N

-LRB-

t

k

-RRB-

model

computer

digital

Term

t

k

N

-LRB-

t

k

-RRB-

model

computer

disk

Total

pages

New

pages

-LRB-

a

-RRB-

After

q

q

-LRB-

b

-RRB-

New

from

q

i

computer

Term

t

k

N

-LRB-

t

k

-RRB-

model

computer

disk

digital

Total

pages

-LRB-

c

-RRB-

After

q

q

i

Figure

Updating

the

query

statistics

table

q

i

q

q

q

i

S

Figure

A

Web

site

that

does

not

return

all

the

results

and

the

keyword

disk

The

table

in

Figure

-LRB-

b

-RRB-

shows

the

frequency

of

each

term

in

the

newly-downloaded

pages

We

can

update

the

old

table

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

to

include

this

new

information

by

simply

adding

corresponding

entries

in

Figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

The

result

is

shown

on

Figure

-LRB-

c

-RRB-

For

example

keyword

model

exists

in

pages

within

the

pages

retrieved

from

q

q

i

According

to

this

new

table

P

-LRB-

model

q

q

i

-RRB-

is

now

Crawling

sites

that

limit

the

number

of

results

In

certain

cases

when

a

query

matches

a

large

number

of

pages

the

Hidden

Web

site

returns

only

a

portion

of

those

pages

For

example

the

Open

Directory

Project

-LSB-

-RSB-

allows

the

users

to

see

only

up

to

results

after

they

issue

a

query

Obviously

this

kind

of

limitation

has

an

immediate

effect

on

our

Hidden

Web

crawler

First

since

we

can

only

retrieve

up

to

a

specific

number

of

pages

per

query

our

crawler

will

need

to

issue

more

queries

-LRB-

and

potentially

will

use

up

more

resources

-RRB-

in

order

to

download

all

the

pages

Second

the

query

selection

method

that

we

presented

in

Section

assumes

that

for

every

potential

query

q

i

we

can

find

P

-LRB-

q

i

q

q

-RRB-

That

is

for

every

query

q

i

we

can

find

the

fraction

of

documents

in

the

whole

text

database

that

contains

q

i

with

at

least

one

of

q

q

However

if

the

text

database

returned

only

a

portion

of

the

results

for

any

of

the

q

q

then

the

value

P

-LRB-

q

i

q

q

-RRB-

is

not

accurate

and

may

affect

our

decision

for

the

next

query

q

i

and

potentially

the

performance

of

our

crawler

Since

we

can

not

retrieve

more

results

per

query

than

the

maximum

number

the

Web

site

allows

our

crawler

has

no

other

choice

besides

submitting

more

queries

However

there

is

a

way

to

estimate

the

correct

value

for

P

-LRB-

q

i

q

q

-RRB-

in

the

case

where

the

Web

site

returns

only

a

portion

of

the

results

Again

assume

that

the

Hidden

Web

site

we

are

currently

crawling

is

represented

as

the

rectangle

on

Figure

and

its

pages

as

points

in

the

figure

Assume

that

we

have

already

issued

queries

q

q

which

returned

a

number

of

results

less

than

the

maximum

number

than

the

site

allows

and

therefore

we

have

downloaded

all

the

pages

for

these

queries

-LRB-

big

circle

in

Figure

-RRB-

That

is

at

this

point

our

estimation

for

P

-LRB-

q

i

q

q

-RRB-

is

accurate

Now

assume

that

we

submit

query

q

i

to

the

Web

site

but

due

to

a

limitation

in

the

number

of

results

that

we

get

back

we

retrieve

the

set

q

i

-LRB-

small

circle

in

Figure

-RRB-

instead

of

the

set

q

i

-LRB-

dashed

circle

in

Figure

-RRB-

Now

we

need

to

update

our

query

statistics

table

so

that

it

has

accurate

information

for

the

next

step

That

is

although

we

got

the

set

q

i

back

for

every

potential

query

q

i

we

need

to

find

P

-LRB-

q

i

q

q

i

-RRB-

P

-LRB-

q

i

q

q

i

-RRB-

P

-LRB-

q

q

i

-RRB-

-LSB-

P

-LRB-

q

i

-LRB-

q

q

-RRB-

-RRB-

P

-LRB-

q

i

q

i

-RRB-

P

-LRB-

q

i

q

i

-LRB-

q

q

-RRB-

-RRB-

-RSB-

-LRB-

-RRB-

In

the

previous

equation

we

can

find

P

-LRB-

q

q

i

-RRB-

by

estimating

P

-LRB-

q

i

-RRB-

with

the

method

shown

in

Section

Additionally

we

can

calculate

P

-LRB-

q

i

-LRB-

q

q

-RRB-

-RRB-

and

P

-LRB-

q

i

q

i

-LRB-

q

q

-RRB-

-RRB-

by

directly

examining

the

documents

that

we

have

downloaded

from

queries

q

q

The

term

P

-LRB-

q

i

q

i

-RRB-

however

is

unknown

and

we

need

to

estimate

it

Assuming

that

q

i

is

a

random

sample

of

q

i

then

P

-LRB-

q

i

q

i

-RRB-

P

-LRB-

q

i

q

i

-RRB-

P

-LRB-

q

i

-RRB-

P

-LRB-

q

i

-RRB-

-LRB-

-RRB-

From

Equation

we

can

calculate

P

-LRB-

q

i

q

i

-RRB-

and

after

we

replace

this

value

to

Equation

we

can

find

P

-LRB-

q

i

q

q

i

-RRB-

In

this

section

we

experimentally

evaluate

the

performance

of

the

various

algorithms

for

Hidden

Web

crawling

presented

in

this

paper

Our

goal

is

to

validate

our

theoretical

analysis

through

real-world

experiments

by

crawling

popular

Hidden

Web

sites

of

textual

databases

Since

the

number

of

documents

that

are

discovered

and

downloaded

from

a

textual

database

depends

on

the

selection

of

the

words

that

will

be

issued

as

queries

to

the

search

interface

of

each

site

we

compare

the

various

selection

policies

that

were

described

in

section

namely

the

random

generic-frequency

and

adaptive

algorithms

The

adaptive

algorithm

learns

new

keywords

and

terms

from

the

documents

that

it

downloads

and

its

selection

process

is

driven

by

a

cost

model

as

described

in

Section

To

keep

our

experiment

and

its

analysis

simple

at

this

point

we

will

assume

that

the

cost

for

every

query

is

constant

That

is

our

goal

is

to

maximize

the

number

of

downloaded

pages

by

issuing

the

least

number

of

queries

Later

in

Section

we

will

present

a

comparison

of

our

policies

based

on

a

more

elaborate

cost

model

In

addition

we

use

the

independence

estimator

-LRB-

Section

-RRB-

to

estimate

P

-LRB-

q

i

-RRB-

from

downloaded

pages

Although

the

independence

estimator

is

a

simple

estimator

our

experiments

will

show

that

it

can

work

very

well

in

practice

For

the

generic-frequency

policy

we

compute

the

frequency

distribution

of

words

that

appear

in

a

corpus

Throughout

our

experiments

once

an

algorithm

has

submitted

a

query

to

a

database

we

exclude

the

query

from

subsequent

submissions

to

the

same

database

from

the

same

algorithm

We

defer

the

reporting

of

results

based

on

the

Zipf

estimation

to

a

future

work

downloaded

from

Web

sites

of

various

topics

-LSB-

-RSB-

Keywords

are

selected

based

on

their

decreasing

frequency

with

which

they

appear

in

this

document

set

with

the

most

frequent

one

being

selected

first

followed

by

the

second-most

frequent

keyword

etc.

Regarding

the

random

policy

we

use

the

same

set

of

words

collected

from

the

Web

corpus

but

in

this

case

instead

of

selecting

keywords

based

on

their

relative

frequency

we

choose

them

ran-domly

-LRB-

uniform

distribution

-RRB-

In

order

to

further

investigate

how

the

quality

of

the

potential

query-term

list

affects

the

random-based

algorithm

we

construct

two

sets

one

with

the

most

frequent

words

of

the

term

collection

used

in

the

generic-frequency

policy

-LRB-

hereafter

the

random

policy

with

the

set

of

words

will

be

referred

to

as

-RRB-

and

another

set

with

the

million

most

frequent

words

of

the

same

collection

as

above

-LRB-

hereafter

referred

to

as

-RRB-

The

former

set

has

frequent

words

that

appear

in

a

large

number

of

documents

-LRB-

at

least

in

our

collection

-RRB-

and

therefore

can

be

considered

of

high-quality

terms

The

latter

set

though

contains

a

much

larger

collection

of

words

among

which

some

might

be

bogus

and

meaningless

The

experiments

were

conducted

by

employing

each

one

of

the

aforementioned

algorithms

-LRB-

adaptive

generic-frequency

and

-RRB-

to

crawl

and

download

contents

from

three

Hidden

Web

sites

The

PubMed

Medical

Library

Amazon

and

the

Open

Directory

Project

-LSB-

-RSB-

According

to

the

information

on

PubMed

's

Web

site

its

collection

contains

approximately

million

abstracts

of

biomedical

articles

We

consider

these

abstracts

as

the

documents

in

the

site

and

in

each

iteration

of

the

adaptive

policy

we

use

these

abstracts

as

input

to

the

algorithm

Thus

our

goal

is

to

discover

as

many

unique

abstracts

as

possible

by

repeat-edly

querying

the

Web

query

interface

provided

by

PubMed

The

Hidden

Web

crawling

on

the

PubMed

Web

site

can

be

considered

as

topic-specific

due

to

the

fact

that

all

abstracts

within

PubMed

are

related

to

the

fields

of

medicine

and

biology

In

the

case

of

the

Amazon

Web

site

we

are

interested

in

downloading

all

the

hidden

pages

that

contain

information

on

books

The

querying

to

Amazon

is

performed

through

the

Software

De-veloper

's

Kit

that

Amazon

provides

for

interfacing

to

its

Web

site

and

which

returns

results

in

XML

form

The

generic

keyword

field

is

used

for

searching

and

as

input

to

the

adaptive

policy

we

extract

the

product

description

and

the

text

of

customer

reviews

when

present

in

the

XML

reply

Since

Amazon

does

not

provide

any

information

on

how

many

books

it

has

in

its

catalogue

we

use

random

sampling

on

the

ISBN

number

of

the

books

to

estimate

the

size

of

the

collection

Out

of

the

random

ISBN

numbers

queried

are

found

in

the

Amazon

catalogue

therefore

the

size

of

its

book

collection

is

estimated

to

be

million

books

It

's

also

worth

noting

here

that

Amazon

poses

an

upper

limit

on

the

number

of

results

-LRB-

books

in

our

case

-RRB-

returned

by

each

query

which

is

set

to

As

for

the

third

Hidden

Web

site

the

Open

Directory

Project

-LRB-

hereafter

also

referred

to

as

dmoz

-RRB-

the

site

maintains

the

links

to

million

sites

together

with

a

brief

summary

of

each

listed

site

The

links

are

searchable

through

a

keyword-search

interface

We

consider

each

indexed

link

together

with

its

brief

summary

as

the

document

of

the

dmoz

site

and

we

provide

the

short

summaries

to

the

adaptive

algorithm

to

drive

the

selection

of

new

keywords

for

querying

On

the

dmoz

Web

site

we

perform

two

Hidden

Web

crawls

the

first

is

on

its

generic

collection

of

indexed

We

did

not

manually

exclude

stop

words

-LRB-

e.g.

the

is

of

etc.

-RRB-

from

the

keyword

list

As

it

turns

out

all

Web

sites

except

PubMed

return

matching

documents

for

the

stop

words

such

as

the

PubMed

Medical

Library

Amazon

Inc.

fraction

of

documents

query

number

Cumulative

fraction

of

unique

documents

PubMed

website

adaptive

generic-frequency

Figure

Coverage

of

policies

for

Pubmed

fraction

of

documents

query

number

Cumulative

fraction

of

unique

documents

Amazon

website

adaptive

generic-frequency

Figure

Coverage

of

policies

for

Amazon

sites

regardless

of

the

category

that

they

fall

into

The

other

crawl

is

performed

specifically

on

the

Arts

section

of

dmoz

-LRB-

http

dmoz.org/Arts

-RRB-

which

comprises

of

approximately

indexed

sites

that

are

relevant

to

Arts

making

this

crawl

topic-specific

as

in

PubMed

Like

Amazon

dmoz

also

enforces

an

upper

limit

on

the

number

of

returned

results

which

is

links

with

their

summaries

Comparison

of

policies

The

first

question

that

we

seek

to

answer

is

the

evolution

of

the

coverage

metric

as

we

submit

queries

to

the

sites

That

is

what

fraction

of

the

collection

of

documents

stored

in

the

Hidden

Web

site

can

we

download

as

we

continuously

query

for

new

words

selected

using

the

policies

described

above

More

formally

we

are

interested

in

the

value

of

P

-LRB-

q

q

q

i

-RRB-

after

we

submit

q

q

i

queries

and

as

i

increases

In

Figures

and

we

present

the

coverage

metric

for

each

policy

as

a

function

of

the

query

number

for

the

Web

sites

of

PubMed

Amazon

general

dmoz

and

the

art-specific

dmoz

respectively

On

the

y-axis

the

fraction

of

the

total

documents

downloaded

from

the

website

is

plotted

while

the

x-axis

represents

the

query

number

A

first

observation

from

these

graphs

is

that

in

general

the

generic-frequency

and

the

adaptive

policies

perform

much

better

than

the

random-based

algorithms

In

all

of

the

figures

the

graphs

for

the

and

the

are

significantly

below

those

of

other

policies

Between

the

generic-frequency

and

the

adaptive

policies

we

can

see

that

the

latter

outperforms

the

former

when

the

site

is

topic

fraction

of

documents

query

number

Cumulative

fraction

of

unique

documents

dmoz

website

adaptive

generic-frequency

Figure

Coverage

of

policies

for

general

dmoz

fraction

of

documents

query

number

Cumulative

fraction

of

unique

documents

dmoz/Arts

website

adaptive

generic-frequency

Figure

Coverage

of

policies

for

the

Arts

section

of

dmoz

cific

For

example

for

the

PubMed

site

-LRB-

Figure

-RRB-

the

adaptive

algorithm

issues

only

queries

to

download

almost

of

the

documents

stored

in

PubMed

but

the

generic-frequency

algorithm

requires

queries

for

the

same

coverage

For

the

dmoz/Arts

crawl

-LRB-

Figure

-RRB-

the

difference

is

even

more

substantial

the

adaptive

policy

is

able

to

download

of

the

total

sites

indexed

in

the

Directory

by

issuing

queries

while

the

frequency-based

algorithm

is

much

less

effective

using

the

same

number

of

queries

and

discovers

only

of

the

total

number

of

indexed

sites

The

adaptive

algorithm

by

examining

the

contents

of

the

pages

that

it

downloads

at

each

iteration

is

able

to

identify

the

topic

of

the

site

as

expressed

by

the

words

that

appear

most

frequently

in

the

result-set

Consequently

it

is

able

to

select

words

for

subsequent

queries

that

are

more

relevant

to

the

site

than

those

preferred

by

the

generic-frequency

policy

which

are

drawn

from

a

large

generic

collection

Table

shows

a

sample

of

keywords

out

of

chosen

and

submitted

to

the

PubMed

Web

site

by

the

adaptive

algorithm

but

not

by

the

other

policies

For

each

keyword

we

present

the

number

of

the

iteration

along

with

the

number

of

results

that

it

returned

As

one

can

see

from

the

table

these

keywords

are

highly

relevant

to

the

topics

of

medicine

and

biology

of

the

Public

Medical

Library

and

match

against

numerous

articles

stored

in

its

Web

site

In

both

cases

examined

in

Figures

and

the

random-based

policies

perform

much

worse

than

the

adaptive

algorithm

and

the

generic-frequency

It

is

worthy

noting

however

that

the

random-based

policy

with

the

small

carefully

selected

set

of

qual-ity

words

manages

to

download

a

considerable

fraction

of

Iteration

Keyword

Number

of

Results

department

patients

clinical

treatment

medical

hospital

disease

protein

Table

Sample

of

keywords

queried

to

PubMed

exclusively

by

the

adaptive

policy

from

the

PubMed

Web

site

after

queries

while

the

coverage

for

the

Arts

section

of

dmoz

reaches

after

queried

keywords

On

the

other

hand

the

random-based

approach

that

makes

use

of

the

vast

collection

of

million

words

among

which

a

large

number

is

bogus

keywords

fails

to

download

even

a

mere

of

the

total

collection

after

submitting

the

same

number

of

query

words

For

the

generic

collections

of

Amazon

and

the

dmoz

sites

shown

in

Figures

and

respectively

we

get

mixed

results

The

generic-frequency

policy

shows

slightly

better

performance

than

the

adaptive

policy

for

the

Amazon

site

-LRB-

Figure

-RRB-

and

the

adaptive

method

clearly

outperforms

the

generic-frequency

for

the

general

dmoz

site

-LRB-

Figure

-RRB-

A

closer

look

at

the

log

files

of

the

two

Hidden

Web

crawlers

reveals

the

main

reason

Amazon

was

functioning

in

a

very

flaky

way

when

the

adaptive

crawler

visited

it

resulting

in

a

large

number

of

lost

results

Thus

we

suspect

that

the

slightly

poor

performance

of

the

adaptive

policy

is

due

to

this

experimental

variance

We

are

currently

running

another

experiment

to

verify

whether

this

is

indeed

the

case

Aside

from

this

experimental

variance

the

Amazon

result

indicates

that

if

the

collection

and

the

words

that

a

Hidden

Web

site

contains

are

generic

enough

then

the

generic-frequency

approach

may

be

a

good

candidate

algorithm

for

effective

crawling

As

in

the

case

of

topic-specific

Hidden

Web

sites

the

random-based

policies

also

exhibit

poor

performance

compared

to

the

other

two

algorithms

when

crawling

generic

sites

for

the

Amazon

Web

site

succeeds

in

downloading

almost

after

issuing

queries

alas

for

the

generic

collection

of

dmoz

the

fraction

of

the

collection

of

links

downloaded

is

after

the

query

Finally

as

expected

is

even

worse

than

downloading

only

of

Amazon

and

of

the

generic

dmoz

In

summary

the

adaptive

algorithm

performs

remarkably

well

in

all

cases

it

is

able

to

discover

and

download

most

of

the

documents

stored

in

Hidden

Web

sites

by

issuing

the

least

number

of

queries

When

the

collection

refers

to

a

specific

topic

it

is

able

to

identify

the

keywords

most

relevant

to

the

topic

of

the

site

and

consequently

ask

for

terms

that

is

most

likely

that

will

return

a

large

number

of

results

On

the

other

hand

the

generic-frequency

policy

proves

to

be

quite

effective

too

though

less

than

the

adaptive

it

is

able

to

retrieve

relatively

fast

a

large

portion

of

the

collection

and

when

the

site

is

not

topic-specific

its

effectiveness

can

reach

that

of

adaptive

-LRB-

e.g.

Amazon

-RRB-

Finally

the

random

policy

performs

poorly

in

general

and

should

not

be

preferred

Impact

of

the

initial

query

An

interesting

issue

that

deserves

further

examination

is

whether

the

initial

choice

of

the

keyword

used

as

the

first

query

issued

by

the

adaptive

algorithm

affects

its

effectiveness

in

subsequent

itera-tions

The

choice

of

this

keyword

is

not

done

by

the

selection

of

the

fraction

of

documents

query

number

Convergence

of

adaptive

under

different

initial

queries

PubMed

website

pubmed

data

information

return

Figure

Convergence

of

the

adaptive

algorithm

using

different

initial

queries

for

crawling

the

PubMed

Web

site

adaptive

algorithm

itself

and

has

to

be

manually

set

since

its

query

statistics

tables

have

not

been

populated

yet

Thus

the

selection

is

generally

arbitrary

so

for

purposes

of

fully

automating

the

whole

process

some

additional

investigation

seems

necessary

For

this

reason

we

initiated

three

adaptive

Hidden

Web

crawlers

targeting

the

PubMed

Web

site

with

different

seed-words

the

word

data

which

returns

results

the

word

information

that

reports

documents

and

the

word

return

that

retrieves

pages

out

of

million

These

keywords

represent

varying

degrees

of

term

popularity

in

PubMed

with

the

first

one

being

of

high

popularity

the

second

of

medium

and

the

third

of

low

We

also

show

results

for

the

keyword

pubmed

used

in

the

experiments

for

coverage

of

Section

and

which

returns

articles

As

we

can

see

from

Figure

after

a

small

number

of

queries

all

four

crawlers

roughly

download

the

same

fraction

of

the

collection

regardless

of

their

starting

point

Their

coverages

are

roughly

equivalent

from

the

query

Eventually

all

four

crawlers

use

the

same

set

of

terms

for

their

queries

regardless

of

the

initial

query

In

the

specific

experiment

from

the

query

onward

all

four

crawlers

use

the

same

terms

for

their

queries

in

each

iteration

or

the

same

terms

are

used

off

by

one

or

two

query

numbers

Our

result

confirms

the

observation

of

-LSB-

-RSB-

that

the

choice

of

the

initial

query

has

minimal

effect

on

the

final

performance

We

can

explain

this

intuitively

as

follows

Our

algorithm

approximates

the

optimal

set

of

queries

to

use

for

a

particular

Web

site

Once

the

algorithm

has

issued

a

significant

number

of

queries

it

has

an

accurate

estimation

of

the

content

of

the

Web

site

regardless

of

the

initial

query

Since

this

estimation

is

similar

for

all

runs

of

the

algorithm

the

crawlers

will

use

roughly

the

same

queries

Impact

of

the

limit

in

the

number

of

results

While

the

Amazon

and

dmoz

sites

have

the

respective

limit

of

and

in

their

result

sizes

these

limits

may

be

larger

than

those

imposed

by

other

Hidden

Web

sites

In

order

to

investigate

how

a

tighter

limit

in

the

result

size

affects

the

performance

of

our

algorithms

we

performed

two

additional

crawls

to

the

generic-dmoz

site

we

ran

the

generic-frequency

and

adaptive

policies

but

we

retrieved

only

up

to

the

top

results

for

every

query

In

Figure

we

plot

the

coverage

for

the

two

policies

as

a

function

of

the

number

of

queries

As

one

might

expect

by

comparing

the

new

result

in

Figure

to

that

of

Figure

where

the

result

limit

was

we

conclude

that

the

tighter

limit

requires

a

higher

number

of

queries

to

achieve

the

same

coverage

For

example

when

the

result

limit

was

the

adaptive

Fraction

of

Unique

Pages

Query

Number

Cumulative

fraction

of

unique

pages

downloaded

per

query

Dmoz

Web

site

-LRB-

cap

limit

-RRB-

adaptive

generic-frequency

Figure

Coverage

of

general

dmoz

after

limiting

the

number

of

results

to

icy

could

download

of

the

site

after

issuing

queries

while

it

had

to

issue

queries

to

download

of

the

site

when

the

limit

was

On

the

other

hand

our

new

result

shows

that

even

with

a

tight

result

limit

it

is

still

possible

to

download

most

of

a

Hidden

Web

site

after

issuing

a

reasonable

number

of

queries

The

adaptive

policy

could

download

more

than

of

the

site

after

issuing

queries

when

the

limit

was

Finally

our

result

shows

that

our

adaptive

policy

consistently

outperforms

the

generic-frequency

policy

regardless

of

the

result

limit

In

both

Figure

and

Figure

our

adaptive

policy

shows

significantly

larger

coverage

than

the

generic-frequency

policy

for

the

same

number

of

queries

Incorporating

the

document

download

cost

For

brevity

of

presentation

the

performance

evaluation

results

provided

so

far

assumed

a

simplified

cost-model

where

every

query

involved

a

constant

cost

In

this

section

we

present

results

regarding

the

performance

of

the

adaptive

and

generic-frequency

algorithms

using

Equation

to

drive

our

query

selection

process

As

we

discussed

in

Section

this

query

cost

model

includes

the

cost

for

submitting

the

query

to

the

site

retrieving

the

result

index

page

and

also

downloading

the

actual

pages

For

these

costs

we

examined

the

size

of

every

result

in

the

index

page

and

the

sizes

of

the

documents

and

we

chose

c

q

c

r

and

c

d

as

values

for

the

parameters

of

Equation

and

for

the

particular

experiment

that

we

ran

on

the

PubMed

website

The

values

that

we

selected

imply

that

the

cost

for

issuing

one

query

and

retrieving

one

result

from

the

result

index

page

are

roughly

the

same

while

the

cost

for

downloading

an

actual

page

is

times

larger

We

believe

that

these

values

are

reasonable

for

the

PubMed

Web

site

Figure

shows

the

coverage

of

the

adaptive

and

generic-frequency

algorithms

as

a

function

of

the

resource

units

used

during

the

download

process

The

horizontal

axis

is

the

amount

of

resources

used

and

the

vertical

axis

is

the

coverage

As

it

is

evident

from

the

graph

the

adaptive

policy

makes

more

efficient

use

of

the

available

resources

as

it

is

able

to

download

more

articles

than

the

generic-frequency

using

the

same

amount

of

resource

units

However

the

difference

in

coverage

is

less

dramatic

in

this

case

compared

to

the

graph

of

Figure

The

smaller

difference

is

due

to

the

fact

that

under

the

current

cost

metric

the

download

cost

of

documents

constitutes

a

significant

portion

of

the

cost

Therefore

when

both

policies

downloaded

the

same

number

of

documents

the

saving

of

the

adaptive

policy

is

not

as

dramatic

as

before

That

Fraction

of

Unique

Pages

Total

Cost

-LRB-

c

q

c

r

c

d

-RRB-

Cumulative

fraction

of

unique

pages

downloaded

per

cost

unit

PubMed

Web

site

adaptive

frequency

Figure

Coverage

of

PubMed

after

incorporating

the

document

download

cost

is

the

savings

in

the

query

cost

and

the

result

index

download

cost

is

only

a

relatively

small

portion

of

the

overall

cost

Still

we

observe

noticeable

savings

from

the

adaptive

policy

At

the

total

cost

of

for

example

the

coverage

of

the

adaptive

policy

is

roughly

while

the

coverage

of

the

frequency

policy

is

only

In

a

recent

study

Raghavan

and

Garcia-Molina

-LSB-

-RSB-

present

an

architectural

model

for

a

Hidden

Web

crawler

The

main

focus

of

this

work

is

to

learn

Hidden-Web

query

interfaces

not

to

generate

queries

automatically

The

potential

queries

are

either

provided

manually

by

users

or

collected

from

the

query

interfaces

In

contrast

our

main

focus

is

to

generate

queries

automatically

without

any

human

intervention

The

idea

of

automatically

issuing

queries

to

a

database

and

examining

the

results

has

been

previously

used

in

different

contexts

For

example

in

-LSB-

-RSB-

Callan

and

Connel

try

to

acquire

an

accurate

language

model

by

collecting

a

uniform

random

sample

from

the

database

In

-LSB-

-RSB-

Lawrence

and

Giles

issue

random

queries

to

a

number

of

Web

Search

Engines

in

order

to

estimate

the

fraction

of

the

Web

that

has

been

indexed

by

each

of

them

In

a

similar

fashion

Bharat

and

Broder

-LSB-

-RSB-

issue

random

queries

to

a

set

of

Search

Engines

in

order

to

estimate

the

relative

size

and

overlap

of

their

indexes

In

-LSB-

-RSB-

Barbosa

and

Freire

experimentally

evaluate

methods

for

building

multi-keyword

queries

that

can

return

a

large

fraction

of

a

document

collection

Our

work

differs

from

the

previous

studies

in

two

ways

First

it

provides

a

theoretical

framework

for

analyzing

the

process

of

generating

queries

for

a

database

and

examining

the

results

which

can

help

us

better

understand

the

effectiveness

of

the

methods

presented

in

the

previous

work

Second

we

apply

our

framework

to

the

problem

of

Hidden

Web

crawling

and

demonstrate

the

efficiency

of

our

algorithms

Cope

et

al.

-LSB-

-RSB-

propose

a

method

to

automatically

detect

whether

a

particular

Web

page

contains

a

search

form

This

work

is

complementary

to

ours

once

we

detect

search

interfaces

on

the

Web

using

the

method

in

-LSB-

-RSB-

we

may

use

our

proposed

algorithms

to

download

pages

automatically

from

those

Web

sites

Reference

-LSB-

-RSB-

reports

methods

to

estimate

what

fraction

of

a

text

database

can

be

eventually

acquired

by

issuing

queries

to

the

database

In

-LSB-

-RSB-

the

authors

study

query-based

techniques

that

can

extract

relational

data

from

large

text

databases

Again

these

works

study

orthogonal

issues

and

are

complementary

to

our

work

In

order

to

make

documents

in

multiple

textual

databases

searchable

at

a

central

place

a

number

of

harvesting

approaches

have

been

proposed

-LRB-

e.g.

OAI

-LSB-

-RSB-

-LSB-

-RSB-

-RRB-

These

approaches

essentially

assume

cooperative

document

databases

that

willingly

share

some

of

their

metadata

and/or

documents

to

help

a

third-party

search

engine

to

index

the

documents

Our

approach

assumes

uncoop-erative

databases

that

do

not

share

their

data

publicly

and

whose

documents

are

accessible

only

through

search

interfaces

There

exists

a

large

body

of

work

studying

how

to

identify

the

most

relevant

database

given

a

user

query

-LSB-

-RSB-

This

body

of

work

is

often

referred

to

as

meta-searching

or

database

selection

problem

over

the

Hidden

Web

For

example

-LSB-

-RSB-

suggests

the

use

of

focused

probing

to

classify

databases

into

a

topical

category

so

that

given

a

query

a

relevant

database

can

be

selected

based

on

its

topical

category

Our

vision

is

different

from

this

body

of

work

in

that

we

intend

to

download

and

index

the

Hidden

pages

at

a

central

location

in

advance

so

that

users

can

access

all

the

information

at

their

convenience

from

one

single

location

Traditional

crawlers

normally

follow

links

on

the

Web

to

discover

and

download

pages

Therefore

they

can

not

get

to

the

Hidden

Web

pages

which

are

only

accessible

through

query

interfaces

In

this

paper

we

studied

how

we

can

build

a

Hidden

Web

crawler

that

can

automatically

query

a

Hidden

Web

site

and

download

pages

from

it

We

proposed

three

different

query

generation

policies

for

the

Hidden

Web

a

policy

that

picks

queries

at

random

from

a

list

of

keywords

a

policy

that

picks

queries

based

on

their

frequency

in

a

generic

text

collection

and

a

policy

which

adaptively

picks

a

good

query

based

on

the

content

of

the

pages

downloaded

from

the

Hidden

Web

site

Experimental

evaluation

on

real

Hidden

Web

sites

shows

that

our

policies

have

a

great

potential

In

particular

in

certain

cases

the

adaptive

policy

can

download

more

than

of

a

Hidden

Web

site

after

issuing

approximately

queries

Given

these

results

we

believe

that

our

work

provides

a

potential

mechanism

to

improve

the

search-engine

coverage

of

the

Web

and

the

user

experience

of

Web

search

Future

Work

We

briefly

discuss

some

future-research

avenues

Multi-attribute

Databases

We

are

currently

investigating

how

to

extend

our

ideas

to

structured

multi-attribute

databases

While

generating

queries

for

multi-attribute

databases

is

clearly

a

more

difficult

problem

we

may

exploit

the

following

observation

to

address

this

problem

When

a

site

supports

multi-attribute

queries

the

site

often

returns

pages

that

contain

values

for

each

of

the

query

attributes

For

example

when

an

online

bookstore

supports

queries

on

title

author

and

isbn

the

pages

returned

from

a

query

typically

contain

the

title

author

and

ISBN

of

corresponding

books

Thus

if

we

can

analyze

the

returned

pages

and

extract

the

values

for

each

field

-LRB-

e.g

title

Harry

Potter

author

J.K.

Rowling

etc

-RRB-

we

can

apply

the

same

idea

that

we

used

for

the

textual

database

estimate

the

frequency

of

each

attribute

value

and

pick

the

most

promising

one

The

main

challenge

is

to

automatically

segment

the

returned

pages

so

that

we

can

identify

the

sections

of

the

pages

that

present

the

values

corresponding

to

each

attribute

Since

many

Web

sites

follow

limited

formatting

styles

in

presenting

multiple

attributes

for

example

most

book

titles

are

preceded

by

the

label

Title

we

believe

we

may

learn

page-segmentation

rules

automatically

from

a

small

set

of

training

examples

Other

Practical

Issues

In

addition

to

the

automatic

query

generation

problem

there

are

many

practical

issues

to

be

addressed

to

build

a

fully

automatic

Hidden-Web

crawler

For

example

in

this

paper

we

assumed

that

the

crawler

already

knows

all

query

interfaces

for

Hidden-Web

sites

But

how

can

the

crawler

discover

the

query

interfaces

The

method

proposed

in

-LSB-

-RSB-

may

be

a

good

starting

point

In

addition

some

Hidden-Web

sites

return

their

results

in

batches

of

say

pages

so

the

user

has

to

click

on

a

next

button

in

order

to

see

more

results

In

this

case

a

fully

automatic

Hidden-Web

crawler

should

know

that

the

first

result

index

page

contains

only

a

partial

result

and

press

the

next

button

automatically

Finally

some

Hidden

Web

sites

may

contain

an

infinite

number

of

Hidden

Web

pages

which

do

not

contribute

much

significant

content

-LRB-

e.g.

a

calendar

with

links

for

every

day

-RRB-

In

this

case

the

Hidden-Web

crawler

should

be

able

to

detect

that

the

site

does

not

have

much

more

new

content

and

stop

downloading

pages

from

the

site

Page

similarity

detection

algorithms

may

be

useful

for

this

purpose

-LSB-

-RSB-

-LSB-

-RSB-

Lexisnexis

-LSB-

-RSB-

The

Open

Directory

Project

-LSB-

-RSB-

E.

Agichtein

and

L.

Gravano

Querying

text

databases

for

efficient

information

extraction

In

ICDE

-LSB-

-RSB-

E.

Agichtein

P.

Ipeirotis

and

L.

Gravano

Modeling

query-based

access

to

text

databases

In

WebDB

-LSB-

-RSB-

Article

on

New

York

Times

Old

Search

Engine

the

Library

Tries

to

Fit

Into

a

Google

World

Available

at

http

June

-LSB-

-RSB-

L.

Barbosa

and

J.

Freire

Siphoning

hidden-web

data

through

keyword-based

interfaces

In

SBBD

-LSB-

-RSB-

M.

K.

Bergman

The

deep

web

Surfacing

hidden

value

http

-LSB-

-RSB-

K.

Bharat

and

A.

Broder

A

technique

for

measuring

the

relative

size

and

overlap

of

public

web

search

engines

In

WWW

-LSB-

-RSB-

A.

Z.

Broder

S.

C.

Glassman

M.

S.

Manasse

and

G.

Zweig

Syntactic

clustering

of

the

web

In

WWW

-LSB-

-RSB-

J.

Callan

M.

Connell

and

A.

Du

Automatic

discovery

of

language

models

for

text

databases

In

SIGMOD

-LSB-

-RSB-

J.

P.

Callan

and

M.

E.

Connell

Query-based

sampling

of

text

databases

Information

Systems

-LRB-

-RRB-

-LSB-

-RSB-

K.

C.-C

Chang

B.

He

C.

Li

and

Z.

Zhang

Structured

databases

on

the

web

Observations

and

implications

Technical

report

UIUC

-LSB-

-RSB-

J.

Cho

N.

Shivakumar

and

H.

Garcia-Molina

Finding

replicated

web

collections

In

SIGMOD

-LSB-

-RSB-

W.

Cohen

and

Y.

Singer

Learning

to

query

the

web

In

AAAI

Workshop

on

Internet-Based

Information

Systems

-LSB-

-RSB-

J.

Cope

N.

Craswell

and

D.

Hawking

Automated

discovery

of

search

interfaces

on

the

web

In

Australasian

conference

on

Database

technologies

-LSB-

-RSB-

T.

H.

Cormen

C.

E.

Leiserson

and

R.

L.

Rivest

Introduction

to

Algorithms

Edition

MIT

Press/McGraw

Hill

-LSB-

-RSB-

D.

Florescu

A.

Y.

Levy

and

A.

O.

Mendelzon

Database

techniques

for

the

world-wide

web

A

survey

SIGMOD

Record

-LRB-

-RRB-

-LSB-

-RSB-

B

He

and

K.

C.-C

Chang

Statistical

schema

matching

across

web

query

interfaces

In

SIGMOD

Conference

-LSB-

-RSB-

P.

Ipeirotis

and

L.

Gravano

Distributed

search

over

the

hidden

web

Hierarchical

database

sampling

and

selection

In

VLDB

-LSB-

-RSB-

P.

G.

Ipeirotis

L.

Gravano

and

M.

Sahami

Probe

count

and

classify

Categorizing

hidden

web

databases

In

SIGMOD

-LSB-

-RSB-

C.

Lagoze

and

H.

V.

Sompel

The

Open

Archives

Initiative

Building

a

low-barrier

interoperability

framework

In

JCDL

-LSB-

-RSB-

S.

Lawrence

and

C.

L.

Giles

Searching

the

World

Wide

Web

Science

-LRB-

-RRB-

-LSB-

-RSB-

V.

Z.

Liu

J.

C.

Richard

C.

Luo

and

and

W.

W.

Chu

Dpro

A

probabilistic

approach

for

hidden

web

database

selection

using

dynamic

probing

In

ICDE

-LSB-

-RSB-

X.

Liu

K.

Maly

M.

Zubair

and

M.

L.

Nelson

OAI

Gateway

Service

for

Web

Crawlers

In

JCDL

-LSB-

-RSB-

B.

B.

Mandelbrot

Fractal

Geometry

of

Nature

W.

H.

Freeman

Co.

-LSB-

-RSB-

A.

Ntoulas

J.

Cho

and

C.

Olston

What

's

new

on

the

web

the

evolution

of

the

web

from

a

search

engine

perspective

In

WWW

-LSB-

-RSB-

A.

Ntoulas

P.

Zerfos

and

J.

Cho

Downloading

hidden

web

content

Technical

report

UCLA

-LSB-

-RSB-

S.

Olsen

Does

search

engine

's

power

threaten

web

's

independence

-LSB-

-RSB-

S.

Raghavan

and

H.

Garcia-Molina

Crawling

the

hidden

web

In

VLDB

-LSB-

-RSB-

G.

K.

Zipf

Human

Behavior

and

the

Principle

of

Least-Effort

Addison-Wesley

Cambridge

MA

query selection
deep web
potential bias
keyword queries
efficiency
hidden web crawling
hidden web
coverage
crawler
adaptive algorithm
accurate language model

keyword selection
adaptive algorithmn
deep web crawler
keyword query