Smartcrawl

A

New

Strategy

For

The

Exploration

Of

The

Hidden

Web

The

way

current

search

engines

work

leaves

a

large

amount

of

information

available

in

the

World

Wide

Web

outside

their

catalogues

This

is

due

to

the

fact

that

crawlers

work

by

following

hyperlinks

and

a

few

other

references

and

ignore

HTML

forms

In

this

paper

we

propose

a

search

engine

prototype

that

can

retrieve

information

behind

HTML

forms

by

automatically

generating

queries

for

them

We

describe

the

architecture

some

implementation

details

and

an

experiment

that

proves

that

the

information

is

not

in

fact

indexed

by

current

search

engines

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval

Retrieval

models

Search

process

H.

-LSB-

Information

Interfaces

and

Presentation

-RSB-

Hy-pertext/Hypermedia

Navigation

Algorithms

Experimentation

The

gigantic

growth

in

content

present

in

the

World

Wide

Web

has

turned

search

engines

into

fundamental

tools

when

the

objective

is

searching

for

information

A

study

in

-LSB-

-RSB-

discovered

that

they

are

the

most

used

source

for

finding

answers

to

questions

positioning

themselves

above

books

for

example

However

a

great

deal

of

relevant

information

is

still

hidden

from

general-purpose

search

engines

like

AlltheWeb.com

or

Google

This

part

of

the

Web

known

as

the

Hidden

Web

-LSB-

-RSB-

the

Invisible

Web

-LSB-

-RSB-

or

the

Deep

Web

-LSB-

-RSB-

is

growing

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

WIDM

November

Washington

DC

USA

Copyright

ACM

constantly

even

more

than

the

visible

Web

to

which

we

are

accustomed

-LSB-

-RSB-

This

happens

because

the

crawler

-LRB-

the

program

that

is

responsible

for

autonomous

navigating

the

web

fetching

pages

-RRB-

used

by

current

search

engines

can

not

reach

this

information

There

are

many

reasons

for

this

to

occur

The

Internet

's

own

dynamics

for

example

ends

up

making

the

index

of

search

engines

obsolete

because

even

the

quickest

crawlers

only

manage

to

access

only

a

small

fraction

each

day

of

the

total

information

available

on

the

Web

The

cost

of

interpretation

of

some

types

of

files

as

for

example

Macromedia

Flash

animations

compressed

files

and

programs

-LRB-

executable

files

-RRB-

could

be

high

not

compensating

for

the

indexing

of

the

little

or

frequently

absent

textual

content

For

this

reason

that

content

is

also

not

indexed

for

the

majority

of

search

engines

Dynamic

pages

also

cause

some

problems

for

indexing

There

are

no

technical

problems

since

this

type

of

page

generates

ordinary

HTML

as

responses

for

its

requests

However

they

can

cause

some

challenges

for

the

crawlers

called

spider

traps

-LSB-

-RSB-

which

can

cause

for

example

the

crawler

to

visit

the

same

page

an

infinite

number

of

times

Therefore

some

search

engines

opt

not

to

index

this

type

of

content

Finally

there

are

some

sites

that

store

their

content

in

databases

and

utilize

HTML

forms

as

an

access

interface

This

is

certainly

the

major

barrier

in

the

exploration

of

the

hidden

Web

and

the

problem

that

has

fewer

implemented

solutions

Nowadays

of

the

commercial

search

engines

that

we

use

explore

this

content

which

is

called

the

Truly

Invisible

Web

-LSB-

-RSB-

Two

fundamental

reasons

make

crawling

the

hidden

Web

a

non-trivial

task

-LSB-

-RSB-

First

is

the

issue

of

scale

Another

study

shows

that

the

hidden

content

is

actually

much

greater

than

what

is

currently

publicly

indexed

-LSB-

-RSB-

As

well

as

this

the

interface

for

access

to

this

information

serves

through

the

HTML

forms

are

projected

to

be

manipulated

and

filled

by

humans

creating

a

huge

problem

for

the

crawlers

In

this

paper

we

propose

a

search

engine

prototype

called

SmartCrawl

which

is

capable

of

automatically

attaining

pages

that

are

not

actually

recoverable

by

current

search

engines

and

that

are

secreted

behind

HTML

forms

The

rest

of

this

article

is

organised

in

the

following

manner

Section

shows

related

work

and

in

the

sequence

we

explain

the

construction

of

HTML

forms

and

how

they

can

be

represented

In

sections

and

we

describe

the

prototype

In

Section

the

experimental

results

are

highlighted

and

finally

in

Section

we

conclude

the

paper

There

are

some

proposals

for

the

automatic

exploration

of

this

hidden

content

Lin

and

Chen

's

solution

-LSB-

-RSB-

aims

to

build

up

a

catalogue

of

small

search

engines

located

in

sites

and

given

the

user

searching

terms

choose

which

ones

are

more

likely

to

answer

them

Once

the

search

engines

are

chosen

by

a

module

called

Search

Engine

Selector

the

user

query

is

redirected

by

filling

the

text

field

of

the

form

The

system

submits

the

keywords

and

waits

for

the

results

that

are

combined

subsequently

and

sent

to

the

users

interface

The

HiWE

-LSB-

-RSB-

is

a

different

strategy

which

aims

to

test

combinations

of

values

for

the

HTML

forms

at

the

moment

of

the

crawling

-LRB-

autonomous

navigation

-RRB-

making

the

indexing

of

the

hidden

pages

possible

Once

a

form

in

a

HTML

page

is

found

the

crawler

makes

several

filing

attempts

analyses

and

indexes

the

results

of

the

obtained

pages

Moreover

the

HiWE

has

a

strategy

to

extract

the

labels

of

HTML

forms

by

rendering

the

page

This

is

very

useful

to

obtain

information

and

classify

forms

and

helps

to

fill

in

its

fields

There

are

other

approaches

that

focus

on

the

data

extraction

Lage

et

al.

-LSB-

-RSB-

claims

to

automatically

generate

agents

to

collect

hidden

Web

pages

by

filling

HTML

forms

In

addition

to

this

Liddle

et

al.

-LSB-

-RSB-

perform

a

more

comprehensive

study

about

form

submissions

and

results

processing

This

study

focus

on

how

valuable

information

can

be

obtained

behind

Web

forms

but

do

not

include

a

crawler

to

fetches

them

HTML

forms

are

frequently

found

on

the

web

and

are

generally

used

for

filtering

a

large

amount

of

information

As

shown

in

Figure

from

a

page

with

one

form

the

user

can

provide

several

pieces

of

data

which

will

be

passed

on

to

a

process

in

the

server

which

generates

the

answer

page

The

current

crawlers

do

not

fill

in

form

fields

with

values

making

them

the

major

barrier

for

exploration

of

the

hidden

Web

In

order

to

achieve

this

it

is

vital

to

extract

several

pieces

of

information

from

the

form

An

HTML

form

can

be

built

on

different

manners

including

various

types

of

fields

such

as

comboboxes

radio

buttons

checkboxes

text

fields

hidden

fields

and

so

on

However

the

data

sent

to

the

server

through

the

Common

Gateway

Interface

-LRB-

CGI

-RRB-

is

represented

by

proper

codified

pairs

-LRB-

name

value

-RRB-

This

way

we

can

characterise

a

form

with

which

has

n

fields

as

a

tuple

F

-LCB-

U

-LRB-

N

V

-RRB-

-LRB-

N

V

-RRB-

-LRB-

N

n

V

n

-RRB-

-RCB-

-LRB-

-RRB-

where

U

is

the

URL

for

the

data

that

has

been

submitted

and

-LRB-

N

n

V

n

-RRB-

are

the

pairs

-LRB-

name

value

-RRB-

-LSB-

-RSB-

However

this

is

a

simplification

since

there

are

much

more

information

associated

with

HTML

forms

An

example

is

the

method

by

which

the

form

data

will

be

sent

to

the

server

that

is

by

HTTP

GET

or

POST

Moreover

some

fields

possess

domain

limitations

-LRB-

e.g.

text

fields

with

a

maximum

size

comboboxes

-RRB-

To

do

an

analysis

of

the

form

and

extract

relevant

information

is

not

an

easy

task

but

the

most

difficult

step

surely

is

to

extract

the

field

's

labels

This

is

because

generally

there

is

not

a

formal

relationship

between

them

in

the

HTML

code

For

example

the

label

for

a

text

field

can

be

placed

above

it

separated

by

a

BR

tag

it

can

be

beside

it

or

it

can

be

inserted

inside

table

cells

All

these

pieces

of

data

are

absolutely

necessary

to

be

extracted

for

surpassing

HTML

forms

and

fetching

the

results

page

The

aim

of

SmartCrawl

is

to

bring

a

strategy

that

allows

a

more

complete

exploration

of

the

hidden

content

To

achieve

this

it

managed

to

generate

values

for

a

largest

number

of

forms

Furthermore

it

has

an

architecture

very

similar

to

current

commercial

search

engines

which

means

it

permits

an

easier

implantation

of

strategies

vastly

used

to

gain

performance

and

scalability

as

presented

by

-LSB-

-RSB-

or

-LSB-

-RSB-

Execution

of

the

Prototype

SmartCrawl

is

above

all

a

search

engine

and

therefore

contains

all

its

essential

components

The

difference

is

in

the

fact

that

each

component

has

adaptations

and

some

extra

features

which

enable

them

to

explore

the

hidden

content

of

the

Web

The

main

goal

is

to

index

only

the

pages

that

potentially

are

in

the

non-explorable

part

of

the

Web

To

extract

the

content

from

behind

these

forms

SmartCrawl

generates

values

for

its

fields

and

submits

them

These

values

are

chosen

in

two

different

moments

in

the

indexing

and

when

an

user

performs

a

search

In

the

indexing

phase

once

it

finds

a

form

the

SmartCrawl

extracts

a

set

of

pieces

of

information

from

it

that

allow

queries

-LRB-

combinations

of

possible

values

for

the

form

-RRB-

to

be

created

New

queries

are

also

generated

when

the

user

performs

a

search

For

this

the

forms

that

are

more

likely

to

answer

to

the

search

receive

the

supplied

keywords

However

contrary

to

the

implementation

of

Lin

and

Chen

-LSB-

-RSB-

the

obtained

results

are

also

scheduled

for

indexing

and

not

only

returned

to

the

user

interface

The

process

of

execution

of

SmartCrawl

is

constituted

in

the

following

steps

-LRB-

-RRB-

finding

the

forms

-LRB-

-RRB-

generating

queries

for

them

-LRB-

-RRB-

going

to

the

results

and

-LRB-

-RRB-

searching

created

indexes

Finding

forms

The

first

step

in

the

execution

process

of

the

SmartCrawl

is

the

creation

of

a

number

of

crawlers

that

work

in

parallel

searching

for

pages

that

include

HTML

forms

Every

page

found

is

then

compressed

and

stored

for

further

analysis

At

this

moment

it

acts

like

a

common

crawler

following

only

links

and

references

to

frames

The

pages

stored

by

the

crawler

are

decompressed

afterwards

by

a

indexing

software

component

which

extracts

pieces

of

information

from

each

of

the

forms

found

and

catalogues

them

Beyond

this

every

page

is

indexed

and

associated

with

the

forms

If

the

same

form

is

found

in

distinct

pages

all

of

them

are

indexed

Nevertheless

there

will

be

only

one

representation

of

the

form

Generating

queries

for

the

forms

Another

component

of

the

indexing

software

is

in

charge

of

generating

values

for

the

encountered

forms

The

generation

of

queries

is

based

on

the

collected

information

about

the

form

and

its

fields

Search

Results

CD

Kinks

Face

To

Face

CD

Kinks

Muswell

Hillbillies

CD

Kinks

Misfits

Processing

-LRB-

Server

-RRB-

CDs

CDs

DVDs

DVDs

Kinks

CDs

CDs

DVDs

DVDs

CDs

CDs

DVDs

DVDs

Media

Title

Artist

Kinks

Figure

A

form

processing

The

first

generated

query

is

always

the

default

that

is

the

one

which

uses

all

the

defined

values

in

the

HTML

code

of

the

form

Next

a

pre-defined

number

k

of

other

possible

combinations

is

generated

To

generate

values

for

the

text

fields

-LRB-

which

possess

an

infinite

domain

-RRB-

a

table

that

stores

a

list

of

values

for

a

data

category

is

consulted

based

on

the

field

label

For

every

generated

query

a

further

visit

is

scheduled

for

the

crawler

The

parameters

-LRB-

set

of

field

names

and

values

-RRB-

are

stored

and

a

new

item

is

added

to

the

queue

of

URLs

that

must

be

visited

by

crawlers

Visiting

the

results

The

crawler

is

in

charge

of

executing

its

second

big

goal

which

is

to

submit

the

scheduled

queries

To

accomplish

this

it

needs

an

extra

feature

the

capacity

to

send

parameters

in

HTTP

requests

using

both

the

GET

and

POST

methods

If

it

perceives

that

an

item

in

the

queue

of

URLs

is

a

query

it

submits

the

parameters

and

analyses

the

HTML

code

obtained

as

a

result

in

the

same

way

that

it

does

to

others

The

page

is

then

compressed

stored

and

associated

with

the

information

of

the

original

query

The

indexing

software

decompress

pages

that

contain

results

of

form

submissions

and

indexes

them

From

this

index

the

classification

and

search

software

finds

results

that

contain

all

the

search

terms

formulated

by

the

user

Searching

for

stored

pages

As

soon

as

the

user

performs

a

search

two

steps

are

exe-cuted

by

the

classification

and

search

software

Firstly

the

indexes

created

by

the

indexing

software

are

consulted

to

find

the

keywords

formed

by

the

user

and

the

results

are

returned

in

an

organised

form

-LRB-

the

most

relevant

come

first

-RRB-

in

an

HTML

page

Subsequently

based

on

these

indexed

pages

which

are

associated

with

the

forms

SmartCrawl

selects

forms

that

are

more

likely

to

answer

to

the

user

's

search

and

generate

new

queries

which

will

be

visited

afterwards

by

the

crawlers

and

indexed

in

the

same

way

by

the

indexing

software

The

architecture

at

the

high

level

of

this

application

is

divided

into

crawler

indexing

software

ranking

and

search

software

and

storage

components

As

we

have

seen

the

crawler

is

responsible

for

obtaining

Web

pages

submitting

queries

and

storing

the

results

The

indexing

software

on

the

other

hand

indexes

the

obtained

pages

and

generates

form

queries

The

ranking

and

search

software

uses

the

indexes

to

answer

searches

made

by

the

user

or

redirects

other

forms

and

storage

components

take

care

of

the

storage

of

all

the

information

used

by

other

components

Figure

shows

how

the

main

components

are

available

and

how

they

interact

with

the

storage

components

that

are

represented

by

the

rounded

boxes

They

are

the

Form

Parser

Form

Inquirer

and

Form

Result

Indexer

of

the

indexing

software

the

Document

Seeker

of

the

ranking

and

search

software

and

the

Crawler

Downloader

of

the

crawler

Two

storage

components

support

the

crawling

URL

Queue

and

URL

List

The

first

is

responsible

for

storing

the

line

of

URLs

that

the

Crawlers

Downloaders

need

to

visit

in

this

way

the

URLs

which

will

serve

as

seeds

to

the

autonomous

are

also

added

in

it

As

soon

as

a

Crawler

Downloader

extracts

links

from

an

HTML

page

it

is

in

this

component

that

the

new

URLs

will

also

be

inserted

for

a

further

visit

The

URL

list

on

the

other

hand

stores

the

URLs

that

have

already

been

visited

allowing

the

Crawler

Downloader

keep

track

of

them

When

the

page

includes

a

form

or

a

result

to

a

query

it

needs

to

be

stored

for

subsequent

indexing

The

crawler

stores

the

compressed

content

in

a

storage

component

named

Warehouse

where

it

is

given

a

number

called

storeId

Two

components

of

the

indexing

software

are

responsible

for

decompressing

these

pages

that

have

been

stored

in

the

Warehouse

The

first

called

the

Form

Parser

extracts

information

from

all

the

forms

contained

within

the

page

and

sends

them

to

the

storage

component

Form

List

where

a

number

called

formId

is

associated

with

every

form

The

Form

Parser

is

also

responsible

for

indexing

the

page

which

contains

forms

and

associating

it

to

each

formId

of

the

forms

contained

within

To

index

these

documents

SmartCrawl

uses

a

technique

called

inverted

index

or

inverted

file

The

Document

List

Wordmatch

and

Lexicon

are

the

three

components

that

carry

out

storing

an

indexed

page

The

Document

List

stores

the

title

a

brief

description

of

each

indexed

page

and

its

docId

The

Lexicon

and

Wordmatch

store

the

inverted

index

itself

The

first

contains

a

list

of

pairs

-LRB-

wordId

word

-RRB-

for

each

one

of

the

words

used

in

indexed

documents

The

second

contains

a

list

of

occurrences

of

the

words

in

the

indexed

documents

and

their

position

-LRB-

offset

-RRB-

in

the

text

Wordmatch

is

formed

therefore

by

the

values

of

docId

wordId

and

offset

The

Form

Inquirer

is

the

indexing

software

component

whose

objective

is

to

generate

queries

for

the

stored

forms

in

the

Form

List

To

generate

values

for

the

text

fields

Form

Inquirer

consults

the

list

of

categories

and

values

through

the

component

Categories

Each

query

generated

is

sent

to

the

Query

List

where

a

queryId

is

associated

to

it

and

a

new

URL

is

added

to

the

URL

Queue

Warehouse

URL

Queue

URL

List

Query

List

Form

List

Categories

Wordmatch

Document

List

Lexicon

Crawler

Downloader

Form

Parser

Form

Result

Indexer

Document

Seeker

New

query

Form

Inquirer

Figure

Architecture

at

a

high

level

The

second

component

that

extracts

compressed

pages

from

the

Warehouse

is

the

Form

Result

Indexer

Its

job

is

too

much

easier

than

the

others

as

its

aim

is

only

to

index

the

pages

that

contain

results

of

submitted

queries

and

to

associate

the

proper

queryId

From

the

indexes

created

and

stored

in

Lexicon

and

Wordmatch

the

Document

Seeker

answers

to

user

searches

Every

document

stored

in

the

in

the

Document

List

possesses

a

formId

associated

with

it

and

optionally

a

queryId

when

the

document

is

the

response

to

a

query

With

these

two

numbers

the

Document

Seeker

consults

the

Form

List

and

the

Query

List

to

obtain

the

necessary

information

about

the

way

it

locates

an

indexed

page

on

the

World

Wide

Web

For

example

a

query

to

a

form

that

points

to

the

URL

using

the

HTTP

GET

method

can

be

represented

by

if

it

possesses

only

one

parameter

with

name

q

and

value

brazil

The

Document

Seeker

should

return

the

result

set

in

an

ordered

form

so

that

the

most

relevant

documents

will

be

taking

first

place

A

simple

solution

for

this

is

to

only

take

into

consideration

the

position

of

the

words

in

the

text

and

the

number

of

occurrences

Considering

o

i

as

the

offset

of

the

i-th

encountered

word

in

the

document

which

is

amongst

the

terms

of

the

user

the

relevance

is

given

by

this

r

n

X

i

-LRB-

o

i

-RRB-

-LRB-

-RRB-

In

equation

we

compute

the

rank

of

a

page

by

summing

the

offsets

of

all

user

search

terms

that

appears

in

the

document

and

then

divide

the

result

by

an

arbitrary

number

-LRB-

in

this

case

-RRB-

so

that

the

most

relevant

entry

receives

the

smallest

number

Another

important

role

of

the

Document

Seeker

is

to

redirect

the

search

terms

supplied

by

the

user

to

some

of

the

several

forms

catalogued

in

the

Form

List

To

accomplish

this

it

looks

in

the

Document

List

for

pages

that

contain

search

engines

-LRB-

forms

with

one

and

only

one

text

field

-RRB-

and

that

possess

in

its

text

words

related

to

the

terms

sought

by

the

user

New

queries

are

then

added

to

the

Query

List

and

new

URLs

are

added

to

the

URL

Queue

to

be

visited

by

the

Crawler

Downloader

Labels

extraction

algorithm

A

very

important

task

is

performed

by

a

secondary

component

called

Form

Extractor

It

is

in

charge

of

extracting

diverse

pieces

of

form

information

present

in

an

HTML

page

To

facilitate

the

content

analysis

of

a

page

the

HTML

code

is

converted

into

a

DOM

tree

provided

by

Cyberneko

Html

Parser

-LSB-

-RSB-

From

the

DOM

tree

the

Form

Extractor

looks

for

nodes

which

represent

forms

and

separate

them

from

the

rest

of

the

tree

Each

of

these

sub-trees

which

encompass

all

the

tags

which

are

positioned

between

the

form

and

the

form

is

submitted

for

processing

Amongst

the

data

which

should

be

obtained

undoubtedly

the

fields

and

their

labels

are

the

most

challenging

ones

In

spite

of

having

a

tag

in

the

HTML

specification

called

label

for

the

declaration

of

a

label

it

is

almost

not

used

and

therefore

we

do

not

possess

a

formal

declaration

of

labels

in

the

HTML

code

The

solution

encountered

was

to

establish

a

standard

that

the

labels

must

have

For

the

Form

Extractor

labels

are

continuous

segments

of

texts

which

use

the

same

format

and

have

the

maximum

of

n

words

and

k

characters

These

values

can

be

defined

in

the

configuration

file

Document

Object

Model

DVDs

CDs

DVDs

Media

Title

Artist

Kinks

-LRB-

a

-RRB-

HTML

form

example

-LCB-

Artist

-RCB-

Label

-LCB-

Title

-RCB-

Label

-LCB-

Media

-RCB-

Label

-LCB-

Submit

Reset

-RCB-

Button

Button

-LCB-

artist

-RCB-

Textfield

-LCB-

title

-RCB-

Textfield

-LCB-

cds

CDs

dvds

DVDs

-RCB-

Checkbox

Label

Checkbox

Label

-LRB-

b

-RRB-

Table

representing

the

position

of

the

elements

of

the

form

Figure

Representing

the

positions

of

the

components

of

an

HTML

form

CheckboxField

Label

Media

Name

media

Options

TextField

Label

Artist

Name

artist

Value

Form

Action

Search.jsp

Request

Method

POST

Fields

TextField

Label

Title

Name

title

Value

SubmitButtonField

Name

Submit

Value

submit

Option

Label

CDs

Value

cd

Option

Label

DVDs

Value

dvd

Figure

An

example

of

a

HTML

form

representation

From

the

sub-tree

which

contains

a

certain

form

information

the

Form

Extractor

generates

a

table

which

represents

the

positioning

of

the

elements

contained

in

it

Figure

shows

an

example

of

the

table

generated

by

a

simple

form

The

table

is

generated

considering

the

nodes

in

the

DOM

tree

which

represent

a

common

HTML

table

If

there

are

more

than

one

defined

table

in

the

HTML

code

similar

representations

are

created

Each

cell

has

a

collection

of

the

form

's

elements

The

third

step

of

the

process

is

to

extract

the

labels

of

each

one

of

the

fields

in

the

form

-LRB-

except

for

hidden

fields

and

buttons

-RRB-

and

generate

an

object-orientated

representation

for

the

forms

To

extract

the

labels

the

Form

Extractor

passes

twice

by

the

generated

table

to

the

form

The

first

time

for

each

field

in

the

form

which

require

a

label

it

is

verified

exactly

what

exists

on

the

left

side

of

the

field

-LRB-

even

in

one

adjacent

cell

-RRB-

If

a

label

is

found

in

this

position

this

label

is

immediately

associated

to

the

field

In

the

second

passage

the

fields

which

still

do

not

have

association

with

labels

are

observed

again

however

this

time

the

search

for

the

label

is

done

in

the

above

cell

For

the

fields

of

the

checkbox

and

combobox

kind

the

treatment

is

special

because

apart

from

the

conventional

label

the

items

which

represent

their

domain

also

have

labels

As

in

the

case

of

DVDs

and

CDs

labels

in

Figure

-LRB-

a

-RRB-

The

domain

labels

are

extracted

from

the

right

and

the

label

of

the

set

of

items

is

obtained

from

the

left

In

the

example

the

checkboxes

are

grouped

by

their

names

which

in

this

case

is

media

The

labels

of

each

item

are

extracted

from

the

right

-LRB-

CDs

and

DVDs

-RRB-

and

the

label

of

the

set

of

checkboxes

is

extracted

from

the

left

of

the

first

field

-LRB-

Media

-RRB-

Figure

shows

how

the

object-oriented

structure

for

the

example

above

would

be

The

list

of

categories

and

values

The

Categories

component

is

in

charged

of

controlling

a

list

of

categories

and

values

which

helps

the

Form

Inquirer

to

generate

values

for

the

text

fields

In

order

to

guarantee

better

results

the

name

of

the

category

is

normalized

before

comparing

to

the

field

label

The

normalization

aims

to

-LRB-

-RRB-

remove

punctuation

-LRB-

leaving

just

the

words

-RRB-

-LRB-

-RRB-

convert

graphic

signing

and

other

special

characters

into

simple

ones

and

-LRB-

-RRB-

remove

stop

words

Stop

words

is

a

concept

given

to

the

words

which

can

be

taken

from

sentences

without

changing

its

meaning

being

largely

used

in

normalization

and

some

search

engines

even

extinguish

these

words

from

their

indexes

Great

part

of

the

Stop

Words

are

prepositions

articles

and

auxiliary

verbs

such

as

the

of

or

is

The

list

of

categories

is

automatically

built

by

the

Form

Parser

Once

it

finds

a

field

with

finite

domain

-LRB-

e.g.

comboboxes

-RRB-

the

values

are

extracted

and

added

to

the

categories

list

associated

to

the

field

's

label

When

the

same

value

is

added

more

than

once

to

a

category

it

gets

more

priority

in

relation

to

the

others

This

is

obtained

by

using

a

number

which

means

the

relevance

of

this

value

in

the

category

It

is

based

on

this

number

that

the

set

of

values

is

put

in

order

before

it

is

repassed

to

the

Query

Inquirer

Therefore

the

most

relevant

values

are

tested

first

in

text

fields

Redirecting

queries

As

mentioned

before

associated

to

each

form

there

is

a

set

of

indexed

pages

where

it

has

been

found

These

pages

allow

the

Document

Seeker

to

choose

the

forms

to

which

the

user

's

search

terms

will

be

redirected

to

In

order

to

choose

amongst

several

forms

two

steps

are

taken

-LRB-

-RRB-

finding

which

words

or

sequences

of

words

are

related

to

the

terms

of

the

user

's

search

and

-LRB-

-RRB-

looking

for

these

words

on

the

pages

which

have

small

search

engines

To

solve

the

first

problem

we

could

use

the

stored

index

itself

However

the

volume

of

indexed

information

is

not

so

large

as

to

provide

a

good

set

of

words

It

was

used

therefore

the

catalogue

of

a

general

purpose

search

engine

called

Gigablast

since

it

implements

data

mining

techniques

and

provides

for

the

searching

terms

a

list

of

words

or

sentences

which

frequently

appear

in

the

returning

documents

From

this

set

of

words

the

Document

Seeker

performs

search

on

pages

which

have

search

engines

looking

for

these

terms

also

putting

them

in

order

according

to

equation

For

the

first

n

selected

search

engines

the

Document

Seeker

creates

new

queries

and

add

them

to

the

queue

so

that

they

can

be

submitted

afterwards

in

the

same

way

done

by

the

Form

Inquirer

The

queries

are

created

by

filling

in

form

's

only

text

field

with

the

searching

terms

of

the

user

and

the

other

fields

with

default

values

Search

form

Results

New

queries

Figure

Interface

for

the

search

of

indexed

documents

The

searching

interface

A

searching

interface

was

build

for

the

purpose

of

carrying

out

the

tests

As

shown

in

Figure

it

is

divided

into

three

parts

The

searching

form

allows

the

user

to

provide

the

terms

of

the

search

Furthermore

it

is

possible

to

choose

what

the

target

of

the

search

is

all

the

pages

only

pages

which

have

forms

or

pages

containing

the

results

of

forms

The

results

are

shown

on

a

list

of

documents

that

contain

all

the

searching

terms

offered

by

the

user

On

pages

that

contain

a

query

associated

to

them

it

is

possible

to

visualise

also

the

parameters

In

the

area

called

new

queries

the

generated

queries

which

have

been

scheduled

for

the

crawler

's

visit

are

displayed

The

tests

which

have

been

carried

out

aim

to

test

some

strategies

used

in

the

implementation

of

the

prototype

hence

some

important

aspects

of

the

system

were

tested

separately

Besides

that

an

analysis

of

the

indexed

content

regarding

its

absence

or

not

in

the

current

searching

engines

was

carried

out

In

order

to

support

the

tests

we

started

up

the

crawlers

and

kept

them

up

until

we

have

thousands

indexed

pages

-LRB-

including

only

pages

with

HTML

forms

and

form

results

-RRB-

It

worth

repeat

that

our

strategy

does

not

index

pages

that

do

not

offer

any

challenge

to

regular

crawlers

Label

extraction

algorithm

evaluation

This

phase

aims

to

evaluate

the

algorithm

used

for

the

extraction

of

the

labels

from

the

form

fields

that

was

de-scribed

in

section

To

do

so

forms

were

manually

observed

and

compared

to

the

information

extracted

by

the

Form

Extractor

For

each

one

of

these

fields

it

was

verified

whether

the

choice

made

by

the

algorithm

was

correct

or

not

From

the

forms

evaluated

of

them

-LRB-

-RRB-

were

not

extracted

The

reason

for

this

is

that

the

HTML

was

malformed

and

the

API

used

for

the

extraction

of

the

DOM

tree

-LRB-

NekoHtml

-LSB-

-RSB-

-RRB-

did

not

manage

to

recover

the

error

This

way

fields

from

the

remaining

forms

were

verified

For

of

them

-LRB-

-RRB-

the

algorithm

extracted

the

label

correctly

making

mistakes

only

in

labels

-LRB-

see

Figure

-RRB-

Labels

extracted

-LRB-

-RRB-

correctly

Labels

extracted

-LRB-

-RRB-

incorrectly

Figure

Fraction

of

labels

extracted

correctly

Some

labels

were

not

extracted

correctly

because

they

did

not

fit

within

the

restrictions

defined

in

section

Another

problem

faced

was

when

the

labels

were

not

defined

inside

the

tag

FORM

In

this

case

they

were

not

present

in

the

sub-tree

analysed

by

the

Form

Extractor

making

its

extraction

impossible

Although

our

solution

did

not

reach

the

HiWE

-LSB-

-RSB-

accuracy

to

extract

labels

we

prove

that

it

is

possible

to

get

very

close

results

without

rendering

the

page

-LRB-

that

consumes

much

computing

resources

-RRB-

Moreover

many

of

the

problems

faced

in

this

experiment

can

be

fixed

without

much

effort

Relevances

of

the

queries

generated

by

the

Document

Seeker

In

order

to

analyse

the

results

obtained

by

the

new

generated

queries

from

the

searches

of

users

search

queries

were

submitted

to

the

prototype

by

using

arbitrary

terms

that

are

commonly

used

in

general

purpose

search

engines

such

as

World

Cup

or

Music

Lyrics

For

each

list

of

queries

generated

by

the

Document

Seeker

the

first

five

ones

were

submitted

and

analysed

manually

totalizing

pages

with

results

Each

page

was

verified

whether

the

query

was

successful

or

not

A

successful

query

is

one

which

has

one

or

more

results

in

it

in

contrast

to

pages

with

no

result

or

pages

that

was

not

considered

a

search

engine

results

page

-LRB-

e.g.

mailing

list

registration

form

-RRB-

Errors

-LRB-

-RRB-

Queries

with

no

results

-LRB-

-RRB-

Successful

queries

-LRB-

-RRB-

Figure

Utilizations

of

the

new

queries

generated

by

Document

Seeker

The

result

obtained

was

that

of

the

submitted

queries

brought

some

results

back

were

not

successful

and

of

the

pages

for

some

reason

could

not

be

recovered

Figure

illustrates

better

the

obtained

utilization

Once

the

most

relevant

queries

are

returned

taking

first

place

it

is

probable

that

with

a

larger

number

of

indexed

pages

we

will

get

better

results

Visibility

of

the

indexed

content

The

implementation

of

SmartCrawl

has

as

aim

the

pages

generated

from

the

filling

of

the

HTML

forms

and

which

are

potentially

part

of

the

hidden

web

Despite

the

fact

that

the

current

commercial

search

engines

do

not

have

an

automatic

mechanism

which

fills

in

the

forms

fields

and

obtain

these

data

as

the

SmartCrawl

does

through

common

links

part

of

this

information

can

be

explored

For

instance

a

page

with

the

results

of

a

form

that

utilizes

the

HTTP

method

GET

can

be

accessed

through

an

usual

link

because

all

its

parameters

can

be

passed

in

the

URL

string

itself

In

addition

to

this

once

the

content

is

stored

in

databases

it

is

not

so

difficult

to

find

pages

that

offer

the

same

information

through

different

interfaces

This

phase

aimed

to

verify

how

much

of

the

indexed

content

can

also

be

accessed

by

Google

In

order

to

do

this

pages

with

results

of

GET

and

POST

forms

were

observed

if

for

one

of

the

reasons

stated

before

they

were

not

indexed

by

this

general

purpose

search

engine

We

found

out

that

of

these

pages

are

not

indexed

by

Google

When

only

queries

which

use

the

method

HTTP

POST

-LRB-

which

are

from

the

total

-RRB-

are

observed

this

number

becomes

even

greater

leaving

just

reachable

by

Google

This

work

proposed

a

search

engine

prototype

which

is

capable

of

handling

with

HTML

forms

as

well

as

filling

them

in

automatically

in

order

to

obtain

information

which

is

un-reachable

by

the

current

search

engines

When

compared

to

other

solutions

as

the

HiWE

-LSB-

-RSB-

and

the

solution

that

redirects

queries

by

Lin

and

Chen

-LSB-

-RSB-

the

SmartCrawl

brings

a

big

differential

which

is

the

ability

of

surpassing

great

number

of

forms

The

mentioned

solutions

have

severe

restrictions

which

directly

affect

the

number

of

forms

that

receives

queries

There

is

a

great

deal

of

work

still

to

be

attached

to

the

solution

for

a

better

exploration

of

the

recovered

content

An

example

of

this

is

that

SmartCrawl

does

not

make

any

analysis

of

the

pages

obtained

as

results

of

the

queries

therefore

indexing

pages

which

contain

errors

and

no

results

The

implementation

of

an

algorithm

which

recognizes

these

pages

would

increase

the

quality

of

the

indexed

data

Besides

a

high

performance

structure

was

not

used

for

the

storage

of

the

indexes

This

resulted

in

slow

searching

and

indexing

A

future

work

will

be

the

implementation

of

a

new

indexing

module

-LSB-

-RSB-

M.

K.

Bergman

The

Deep

Web

Surfacing

Hidden

Value

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

anatomy

of

a

large-scale

hypertextual

Web

search

engine

Computer

Networks

and

ISDN

Systems

-LRB-

-RRB-

-LSB-

-RSB-

A.

Clark

Cyberneko

html

parser

andyc

-LSB-

-RSB-

J.

Lage

A.

Silva

P.

Golgher

and

A.

Laender

Collecting

hidden

web

pages

for

data

extraction

In

Proceedings

of

the

ACM

International

Workshop

on

Web

Information

and

Data

Management

-LSB-

-RSB-

S.

Liddle

D.

Embley

D.

Scott

and

S.

H.

Yau

Extracting

data

behind

web

forms

In

Proceedings

of

the

Workshop

on

Conceptual

Modeling

Approaches

for

e-Business

pages

-LSB-

-RSB-

K.-I

Lin

and

H.

Chen

Automatic

information

discovery

from

the

invisible

web

In

Proceedings

of

the

The

International

Conference

on

Information

Technology

Coding

and

Computing

-LRB-

ITCC

-RRB-

pages

-LSB-

-RSB-

S.

Raghavan

and

H.

Garcia-Molina

Crawling

the

hidden

web

In

Proceedings

of

the

International

Conference

on

Very

Large

Databases

pages

-LSB-

-RSB-

A.

Rappoport

Checklist

for

search

robot

crawling

and

indexing

html

-LSB-

-RSB-

C.

Sherman

and

G.

Price

The

Invisible

Web

Uncovering

Information

Sources

Search

Engines

Ca

n't

See

CyberAge

Books

-LSB-

-RSB-

V.

Shkapenyuk

and

T.

Suel

Design

and

implementation

of

a

high-performance

distributed

web

crawler

In

Proceedings

of

the

International

Conference

on

Data

Engineering

pages

-LSB-

-RSB-

D.

Sullivan

Internet

Top

Information

Resource

Study

Finds

B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

search engine
smartcrawl
label extraction
extraction algorithm
experimentation
implementation
architecture
html form
hidden web
web crawler

information retrieval
hidden web content