Integrating

the

Document

Object

Model

with

Hyperlinks

for

Enhanced

Topic

Distillation

and

Information

Extraction

Topic

distillation

is

the

process

of

finding

authoritative

Web

pages

and

comprehensive

hubs

which

reciprocally

endorse

each

other

and

are

relevant

to

a

given

query

Hyperlink-based

topic

distillation

has

been

traditionally

applied

to

a

macroscopic

Web

model

where

documents

are

nodes

in

a

directed

graph

and

hyperlinks

are

edges

Macroscopic

models

miss

valuable

clues

such

as

banners

navigation

panels

and

template-based

inclusions

which

are

embedded

in

HTML

pages

using

markup

tags

Consequently

results

of

macroscopic

distillation

algorithms

have

been

deteriorating

in

quality

as

Web

pages

are

becoming

more

complex

We

propose

a

uniform

fine-grained

model

for

the

Web

in

which

pages

are

represented

by

their

tag

trees

-LRB-

also

called

their

Document

Object

Models

or

DOMs

-RRB-

and

these

DOM

trees

are

interconnected

by

ordinary

hyperlinks

Surprisingly

macroscopic

distillation

algorithms

do

not

work

in

the

fine-grained

scenario

We

present

a

new

algorithm

suitable

for

the

fine-grained

model

It

can

dis-aggregate

hubs

into

coherent

regions

by

segmenting

their

DOMtrees

M

utual

endorsement

between

hubs

and

authorities

involve

these

regions

rather

than

single

nodes

representing

complete

hubs

Anecdotes

and

measurements

using

a

benchmark

suite

used

in

earlier

topic

distillation

research

reveal

two

benefits

from

the

new

algorithm

distillation

quality

improves

and

a

by-product

of

distillation

is

the

ability

to

extract

relevant

snippets

from

hubs

which

are

only

partially

relevant

to

the

query

Kleinberg

's

Hyperlink

Induced

Topic

Search

-LRB-

HITS

-RRB-

-LSB-

-RSB-

and

the

PageRank

algorithm

-LSB-

-RSB-

underlying

Google

have

revolutionized

ranking

technology

for

Web

search

engines

PageRank

evaluates

the

prestige

score

of

a

page

as

roughly

proportional

to

the

sum

of

prestige

scores

of

pages

citing

it

-LRB-

Note

To

view

the

HTML

version

using

Netscape

add

the

following

line

to

your

Xdefaults

or

Xresources

file

Netscape

documentFonts.charset

adobe-fontspecific

For

printing

use

the

PDF

version

as

browsers

may

not

print

the

mathematics

properly

-RRB-

Copyright

is

held

by

author/owner

May

Hong

Kong

using

hyperlinks

HITS

also

identifies

collections

of

resource

links

or

hubs

densely

coupled

to

authoritative

pages

on

a

topic

The

model

of

the

Web

underlying

these

and

related

systems

is

a

directed

graph

with

pages

-LRB-

HTML

files

-RRB-

as

nodes

and

hyperlinks

as

edges

Since

those

papers

were

published

the

Web

has

been

evolving

in

fascinating

ways

apart

from

just

getting

larger

Web

pages

are

changing

from

static

files

to

dynamic

views

generated

from

complex

templates

and

backing

semi-structured

databases

A

variety

of

hypertext-specific

idioms

such

as

navigation

panels

advertisement

banners

link

exchanges

and

Web-rings

have

been

emerging

There

is

also

a

migration

of

Web

content

from

syntac-tic

HTML

markups

towards

richly

tagged

semi-structured

XML

documents

-LRB-

-RRB-

interconnected

at

the

XML

element

level

by

semantically

rich

links

-LRB-

see

e.g.

the

XLink

proposal

at

-RRB-

These

refinements

are

welcome

steps

to

implementing

what

Berners-Lee

and

others

call

the

semantic

Web

-LRB-

-RRB-

but

result

in

document

file

and

site

boundaries

losing

their

traditional

significance

Continual

experiments

performed

by

several

researchers

-LSB-

-RSB-

reveal

a

steady

deterioration

of

distillation

quality

through

the

last

few

years

In

our

experience

poor

results

are

frequently

traced

to

the

following

causes

Links

have

become

more

frequent

and

noisy

from

the

perspective

of

the

query

such

as

in

banners

navigation

panels

and

advertisements

Noisy

links

do

not

carry

human

editorial

endorsement

a

basic

assumption

in

topic

distillation

Hubs

may

be

mixed

meaning

only

a

portion

of

the

hub

may

be

relevant

to

the

query

Macroscopic

distillation

algorithms

treat

whole

pages

as

atomic

indivisible

nodes

with

no

internal

structure

This

leads

to

false

reinforcements

and

resulting

contamination

of

the

query

responses

Thanks

in

part

to

the

visibility

of

Google

content

creators

are

well

aware

of

hyperlink-based

ranking

technology

One

reaction

has

been

the

proliferation

of

nepotistic

clique

attacks

a

collection

of

sites

linking

to

each

other

without

semantic

reason

e.g.

http

and

-LRB-

Figures

and

provide

some

examples

-RRB-

Some

examples

look

suspiciously

like

a

conscious

attempt

to

spam

search

engines

that

use

link

analysis

Interestingly

in

most

cases

the

visual

presentation

clearly

marks

noisy

links

which

surfers

rarely

follow

but

macroscopic

algorithms

are

unable

to

exploit

it

html

head

title

Portals

title

head

body

ul

li

a

href

Yahoo

a

li

li

a

href

Lycos

a

li

ul

body

html

html

head

body

title

ul

li

li

a

a

Figure

In

the

fine-grained

model

DOMs

for

individual

pages

are

trees

interconnected

by

ordinary

hyperlinks.Each

triangle

is

the

DOM

tree

corresponding

to

one

HTML

page.Green

boxes

represent

text

Many

had

hoped

that

HITS-like

algorithms

would

put

an

end

to

spamming

but

clearly

the

situation

is

more

like

an

ongoing

arms-race

Google

combines

link-based

ranking

with

page

text

and

anchor

text

in

undisclosed

ways

and

keeps

tweaking

the

combination

but

suffers

an

occasional

embarrassment

Distillation

has

always

been

observed

to

work

well

for

broad

topics

-LRB-

for

which

there

exist

well-connected

relevant

Web

subgraphs

and

pure

hubs

-RRB-

and

not

too

well

for

narrow

topics

because

w.r.t.

narrow

topics

most

hubs

are

mixed

and

have

too

many

irrelevant

links

Mixed

hubs

and

the

arbitrariness

of

page

boundaries

have

been

known

to

produce

glitches

in

the

Clever

system

-LSB-

-RSB-

there

has

been

no

reliable

way

to

classify

hubs

as

mixed

or

pure

If

a

fine-grained

model

can

suitably

dis-aggregate

mixed

hubs

distillation

should

become

applicable

to

narrow

queries

too

Yet

another

motivation

for

the

fine-grained

model

comes

from

the

proliferation

of

mobile

clients

such

as

cell-phones

and

PDAs

with

small

or

no

screens

Even

on

a

conventional

Web

browser

scrolling

through

search

results

for

promising

responses

then

scrolling

through

those

responses

to

satisfy

a

specific

information

need

are

tedious

steps

The

tedium

is

worse

on

mobile

clients

Search

engines

that

need

to

serve

mobile

clients

must

be

able

to

pinpoint

narrow

sections

of

pages

and

sites

that

address

a

specific

information

need

and

limit

the

amount

of

extra

matter

sent

back

to

the

client

-LSB-

-RSB-

Our

contributions

We

initiate

a

study

of

topic

distillation

with

a

fine-grained

model

of

the

Web

built

using

the

Document

Object

Model

-LRB-

DOM

-RRB-

of

HTML

pages

The

DOM

can

model

reasonably

clean

HTML

support

XML

documents

that

adhere

to

rigid

schema

definitions

and

embed

free

text

in

a

natural

way

In

our

model

HTML

pages

are

represented

by

their

DOMs

and

these

DOMtrees

are

interconnected

by

ordinary

hyperlinks

-LRB-

figure

-RRB-

The

sometimes

artificial

distinction

between

Web-level

site-level

page-level

and

intra-page

structures

is

thereby

blurred

Surprisingly

macroscopic

distillation

algorithms

perform

poorly

in

the

fine-grained

setting

we

demonstrate

this

using

analysis

and

anecdotes

Our

main

technical

contribution

is

a

new

fine-grained

distillation

-LRB-

local

copy

GoogleDrEvil.html

-RRB-

and

-LRB-

local

copy

GoogleBush.html

-RRB-

provide

some

samples

Bibliometry

Graph

theory

PageRank

Google

HITS

Clever@IBM

Exploiting

anchor

text

Topic

distillation

@Compaq

Outlier

elimination

DOM

structure

This

paper

Figure

This

work

in

the

context

of

HITS

and

related

research

gorithm

which

can

identify

mixed

hubs

and

segment

their

corresponding

DOMtrees

into

maximal

subtrees

which

are

coherent

w.r.t.

the

query

i.e.

each

is

almost

completely

relevant

or

completely

irrelevant

The

segmentation

algorithm

uses

the

Minimum

Description

Length

-LRB-

MDL

-RRB-

principle

-LSB-

-RSB-

from

Information

Theory

-LSB-

-RSB-

Rather

than

collapse

these

diverse

hub

subtrees

into

one

node

the

new

algorithm

allocates

a

node

for

each

subtree

This

intermediate

level

of

detail

between

the

macroscopic

and

the

fine-grained

model

is

essential

to

the

success

of

our

algorithm

We

report

on

experiments

with

queries

involving

over

Web

pages

This

benchmark

has

been

used

in

previous

research

on

resource

compilation

and

topic

distillation

-LSB-

-RSB-

Our

experience

is

that

the

fine-grained

model

and

algorithm

significantly

improve

the

quality

of

distillation

and

are

capable

of

extracting

DOMsubtrees

from

mixed

hubs

that

are

relevant

to

the

query

We

note

that

in

this

study

we

have

carefully

and

deliberately

isolated

the

model

from

possible

influences

of

text

analysis

By

controlling

our

experimental

environment

to

not

use

text

we

push

HITS-like

ideas

to

the

limit

evaluating

exactly

the

value

added

by

information

present

in

DOM

structures

In

ongoing

work

we

have

added

textual

support

to

our

framework

and

obtained

even

better

results

-LSB-

-RSB-

Benefits

and

applications

Apart

from

offering

a

more

faithful

model

of

Web

content

our

approach

enables

solutions

to

the

following

problems

Better

topic

distillation

We

show

less

tendency

for

topic

drift

and

contamination

when

the

fine-grained

model

is

used

Web

search

using

devices

with

small

or

no

screen

The

ability

to

identify

page

snippets

relevant

to

a

query

is

attractive

to

search

services

suitable

for

mobile

clients

Focused

crawling

Identification

of

relevant

DOMsubtrees

can

be

used

to

better

guide

a

focused

crawler

's

link

expansion

-LSB-

-RSB-

Annotation

extraction

Experiments

with

a

previous

macroscopic

distillation

algorithm

-LRB-

Clever

-LSB-

-RSB-

-RRB-

revealed

that

volunteers

preferred

Clever

to

Yahoo!

only

when

Yahoo!

's

manual

site

annotations

were

removed

in

a

blind

test

Our

work

may

improve

on

current

techniques

for

automatic

annotation

extraction

-LSB-

-RSB-

by

first

collecting

candidate

hub

page

fragments

and

then

subjecting

the

text

therein

to

further

segmentation

techniques

Data

preparation

for

linguistic

analysis

Information

extraction

is

a

natural

next

step

after

resource

discovery

It

is

easier

to

build

extractors

based

on

statistical

and

linguistic

models

if

the

domain

or

subject

matter

of

the

input

documents

is

suitably

segmented

-LSB-

-RSB-

as

is

effected

by

our

hub

subtree

extraction

technique

which

is

a

natural

successor

to

resource

discovery

and

a

precursor

to

linguistic

analysis

Outline

of

the

paper

In

we

review

HITS

and

related

algorithms

This

section

can

be

skipped

by

a

reader

who

is

familiar

with

HITS-related

literature

In

we

illustrate

some

recent

and

growing

threats

to

the

continued

success

of

macroscopic

distillation

algorithms

We

show

why

the

fine-grained

model

does

not

work

with

traditional

HITS-like

approaches

in

and

then

propose

our

framework

in

We

report

on

experimental

results

in

and

conclude

in

with

some

comments

on

ongoing

and

future

work

We

review

the

HITS

family

of

algorithms

and

discuss

how

they

were

continually

enhanced

to

address

evolving

Web

content

Review

of

HITS

and

related

systems

The

HITS

algorithm

-LSB-

-RSB-

started

with

a

query

q

which

was

sent

to

a

text

search

engine

The

returned

set

of

pages

R

q

was

fetched

from

the

Web

together

with

any

pages

having

a

link

to

any

page

in

R

q

as

well

as

any

page

cited

in

some

page

of

R

q

using

a

hyperlink

Links

that

connected

pages

on

the

same

Web

server

-LRB-

based

on

canonical

host

name

match

-RRB-

were

dropped

from

consideration

because

they

were

often

seen

to

serve

only

a

navigational

purpose

or

were

nepotistic

in

nature

Suppose

the

resulting

graph

is

G

q

-LRB-

V

q

E

q

-RRB-

We

will

drop

the

subscript

q

where

clear

from

context

Each

node

v

in

V

is

assigned

two

scores

the

hub

score

h

-LRB-

v

-RRB-

and

the

authority

score

a

-LRB-

v

-RRB-

initialized

to

any

positive

number

Next

the

HITS

algorithm

alternately

updates

a

and

h

as

follows

a

-LRB-

v

-RRB-

-LRB-

u

v

-RRB-

E

h

-LRB-

u

-RRB-

and

h

-LRB-

u

-RRB-

-LRB-

u

v

-RRB-

E

a

-LRB-

v

-RRB-

making

sure

after

each

iteration

to

scale

a

and

h

so

that

v

h

-LRB-

v

-RRB-

v

a

-LRB-

v

-RRB-

until

the

ranking

of

nodes

by

a

and

h

stabilize

-LRB-

see

figure

-RRB-

If

E

is

represented

in

the

adjacency

matrix

format

-LRB-

i.e.

E

-LSB-

i

j

-RSB-

if

there

is

an

edge

-LRB-

i

j

-RRB-

and

otherwise

-RRB-

then

the

above

operation

can

be

written

simply

as

a

E

T

h

and

h

Ea

interspersed

with

scaling

to

set

h

a

The

HITS

algorithm

effectively

uses

power

iterations

-LSB-

-RSB-

to

find

a

the

principal

eigenvector

of

E

T

E

and

h

the

principal

eigenvector

of

EE

T

Pages

with

large

a

are

popular

or

authoritative

sources

of

information

pages

with

large

h

are

good

collections

of

links

A

key

feature

of

HITS

is

how

endorsement

or

popularity

diffuses

to

siblings

If

-LRB-

u

v

-RRB-

and

-LRB-

u

w

-RRB-

are

edges

and

somehow

a

-LRB-

v

-RRB-

becomes

large

then

in

the

next

iteration

h

-LRB-

u

-RRB-

will

increase

and

in

the

following

iteration

a

-LRB-

w

-RRB-

will

increase

We

will

describe

this

as

v

's

authority

diffuses

to

w

through

the

hub

u.

This

is

how

sibling

nodes

reinforce

each

other

's

authority

scores

We

will

revisit

this

property

later

in

Google

has

no

notion

of

hubs

Roughly

speaking

each

page

v

has

a

single

prestige

score

p

-LRB-

v

-RRB-

called

its

PageRank

-LSB-

-RSB-

which

is

defined

as

proportional

to

-LRB-

u

v

-RRB-

E

p

-LRB-

u

-RRB-

the

sum

of

prestige

scores

of

pages

u

that

cite

v

Some

conjecture

that

the

prestige

model

is

adequate

for

the

living

Web

because

good

hubs

readily

acquire

high

prestige

as

well

Our

work

establishes

the

value

of

a

bipartite

model

like

HITS

and

indeed

the

value

of

an

asymmetric

model

where

hubs

Expanded

graph

Rootset

Keyword

Search

engine

Query

a

Eh

h

E

T

a

h

a

h

h

h

a

a

a

hello

world

stdio

Centroid

of

rootset

Similarity

cone

Distant

vectors

pruned

Figure

-LRB-

a

-RRB-

HITS

a

macroscopic

topic

distillation

algorithm

with

uniform

edge

weights

-LRB-

b

-RRB-

The

B&H

algorithm

apart

from

using

non-uniform

edge

weights

discards

pages

in

the

expanded

set

which

are

too

dissimilar

to

the

rootset

pages

to

prevent

topic

drift.Documents

are

represented

as

vectors

with

each

component

representing

one

token

or

word

-LSB-

-RSB-

are

analyzed

quite

differently

from

authorities

Therefore

we

will

not

discuss

prestige-based

models

any

further

The

impact

of

the

evolving

Web

on

hyperlink

analysis

Elegant

as

the

HITS

model

is

it

does

not

adequately

capture

various

idioms

of

Web

content

We

discuss

here

a

slew

of

follow-up

work

that

sought

to

address

these

issues

Kleinberg

dropped

links

within

the

same

Web-site

from

consideration

because

these

were

often

found

to

be

navigational

nepotistic

and

noisy

Shortly

after

HITS

was

published

Bharat

and

Henzinger

-LRB-

B&H

-LSB-

-RSB-

-RRB-

found

that

nepotism

was

not

limited

to

same-site

links

In

many

trials

with

HITS

they

found

two

distinct

sites

s

and

s

where

s

hosted

a

number

of

pages

u

linking

to

a

page

v

on

s

driving

up

a

-LRB-

v

-RRB-

beyond

what

may

be

considered

fair

B&H

proposed

a

simple

and

effective

fix

for

such

site-pair

nepotism

if

k

pages

on

s

point

to

v

let

the

weight

of

each

of

these

links

be

k

so

that

they

add

up

to

one

assuming

a

site

-LRB-

not

a

page

-RRB-

is

worth

one

unit

of

voting

power

Later

work

in

the

Clever

system

-LSB-

-RSB-

used

a

small

edge

weight

for

same-site

links

and

a

larger

weight

for

other

links

but

these

weights

were

tuned

empirically

by

evaluating

the

results

on

specific

queries

Another

issue

with

HITS

were

mixed

hubs

or

pages

u

that

included

a

collection

of

links

of

which

only

a

subset

was

relevant

to

a

query

Because

HITS

modeled

u

as

a

single

node

with

a

single

h

score

high

authority

scores

could

diffuse

from

relevant

links

to

less

relevant

links

E.g.

responses

to

the

query

movie

awards

sometimes

drifted

into

the

neighboring

more

densely

linked

domain

of

movie

companies

Later

versions

of

Clever

tried

to

address

the

issue

in

two

ways

First

links

within

a

fixed

number

of

tokens

of

query

terms

were

assigned

a

large

edge

weight

-LRB-

the

width

of

the

activation

window

was

tuned

by

trial-and-error

-RRB-

Second

hubs

which

were

too

long

were

segmented

at

a

few

prominent

boundaries

-LRB-

such

as

UL

or

HR

-RRB-

into

pagelets

with

their

own

scores

The

boundaries

were

chosen

using

a

static

set

of

rules

depending

on

the

markup

tags

on

those

pages

alone

To

avoid

drift

B&H

also

computed

a

vector

space

representation

-LSB-

-RSB-

of

documents

in

the

response

set

-LRB-

shown

in

Figure

-RRB-

and

then

dropped

pages

that

were

judged

to

be

outliers

using

a

suitable

threshold

of

-LRB-

cosine

-RRB-

similarity

to

the

vector

space

centroid

B&H

is

effective

for

improving

precision

but

may

reduce

recall

if

mixed

hubs

are

pruned

because

of

small

similarity

to

the

root

set

centroid

This

Query

term

Activation

window

Figure

Clever

uses

a

slightly

more

detailed

page

model

than

HITS.Hyperlinks

near

query

terms

are

given

heavier

weights

Such

links

are

shown

as

thicker

lines

may

in

turn

distort

hub

and

authority

scores

and

hence

the

desired

ranking

Losing

a

few

hubs

may

not

be

a

problem

for

broad

queries

but

could

be

serious

for

narrower

queries

As

resource

discovery

and

topic

distillation

become

more

commonplace

we

believe

the

quest

will

be

for

every

additional

resource

than

can

possibly

be

harvested

not

merely

the

ones

that

leap

out

at

the

surfer

Our

goal

should

therefore

be

to

extract

relevant

links

and

annotations

even

from

pages

which

are

partially

or

largely

irrelevant

HTML

documents

have

always

embedded

many

sources

of

information

-LRB-

other

that

text

-RRB-

which

have

been

largely

ignored

in

previous

distillation

research

Markups

are

one

such

source

From

a

well-formed

HTML

document

it

ought

to

be

possible

to

extract

a

tree

structure

called

the

Document

Object

Model

-LRB-

DOM

-RRB-

In

real

life

HTML

is

rarely

well

formed

but

using

a

few

simple

patches

it

is

possible

to

generate

reasonably

accurate

DOMs

For

XML

sources

adhering

to

a

published

DTD

a

DOMis

precise

and

well

defined

For

simplicity

we

shall

work

with

a

greatly

pared-down

version

of

the

DOMfor

HTML

pages

We

will

discard

all

text

and

only

retain

those

paths

in

the

DOMtree

that

lead

from

the

root

to

a

leaf

which

is

an

A.

element

with

an

HREF

leading

to

another

page

Hyperlinks

always

originate

from

leaf

DOMelements

typically

deep

in

the

DOMtree

of

the

source

document

If

same-site

links

are

ignored

very

few

macro-level

hyperlinks

target

an

internal

node

in

a

DOMtree

-LRB-

using

the

modifier

in

the

URL

-RRB-

To

simplify

our

model

-LRB-

and

experiments

-RRB-

we

will

assume

that

the

target

of

a

hyperlink

is

always

the

root

node

of

a

DOMtree

In

our

experiments

we

found

very

few

URLs

to

be

otherwise

A

first-cut

approach

-LRB-

which

one

may

call

MicroHITS

-RRB-

would

be

to

use

the

fine-grained

graph

directly

in

the

HITS

algorithm

One

may

even

generalize

same-site

to

same-DOM

and

use

B&H

like

edge-weights

This

approach

turns

out

to

work

rather

poorly

To

appreciate

why

consider

two

simple

example

graphs

shown

in

Figure

and

their

associated

eigenvectors

The

first

graph

is

for

the

macro

setting

Expanding

out

a

E

T

Ea

we

get

a

-LRB-

-RRB-

a

-LRB-

-RRB-

a

-LRB-

-RRB-

and

Bipar

tite

c

o

r

e

DOM

Tree

E

E

E

T

E

E

E

T

Figure

A

straight-forward

application

of

HITS-like

algorithms

to

a

DOM

graph

may

result

in

some

internal

DOM

nodes

blocking

the

diffusion

of

authority

across

siblings

a

-LRB-

-RRB-

a

-LRB-

-RRB-

a

-LRB-

-RRB-

which

demonstrates

the

mutual

reinforcement

In

the

second

example

nodes

numbered

and

are

part

of

one

DOM

tree

This

time

we

get

a

-LRB-

-RRB-

-LRB-

-RRB-

a

-LRB-

-RRB-

and

a

-LRB-

-RRB-

a

-LRB-

-RRB-

a

-LRB-

-RRB-

but

there

is

no

coupling

between

a

-LRB-

-RRB-

and

a

-LRB-

-RRB-

which

we

would

expect

at

the

macroscopic

level

Node

-LRB-

marked

red

-RRB-

effectively

blocks

the

authority

from

diffusing

between

nodes

and

One

may

hope

that

bigger

DOMtrees

and

multiple

paths

to

authorities

might

alleviate

the

problem

but

the

above

example

really

depicts

a

basic

problem

The

success

of

HITS

depends

critically

on

reinforcement

among

bipartite

cores

-LRB-

see

figure

-RRB-

which

may

be

destroyed

by

the

introduction

of

fine-grained

nodes

At

this

point

the

dilemma

is

clear

by

collapsing

hubs

into

one

node

macroscopic

distillation

algorithms

lose

valuable

detail

but

the

more

faithful

fine-grained

model

prevents

bipartite

reinforcement

In

this

section

we

present

our

new

model

and

distillation

algorithm

that

resolves

the

dilemma

Informally

our

model

of

hub

generation

enables

our

algorithm

to

find

a

cut

or

frontier

across

each

hub

's

DOMtree

Subtrees

attached

to

these

cuts

are

made

individual

nodes

in

the

distillation

graph

Thus

the

hub

score

of

the

entire

page

is

dis-aggregated

at

this

intermediate

level

The

frontiers

are

not

computed

one

time

as

a

function

of

the

page

alone

neither

do

they

remain

unchanged

during

the

HITS

iterations

The

frontiers

are

determined

by

the

current

estimates

of

the

hub

scores

of

the

leaf

HREF

nodes

We

will

first

describe

the

hub

segmentation

technique

and

then

use

it

in

a

modified

iterative

distillation

algorithm

Scoring

internal

micro-hub

nodes

Macroscopic

distillation

algorithms

rank

and

report

complete

hub

pages

even

if

they

are

only

partially

relevant

In

this

section

we

address

the

problem

of

estimating

the

hub

score

of

each

DOMnode

in

the

fine-grained

graph

given

an

estimate

of

authority

scores

Because

inter-page

hyperlinks

originate

in

leaf

DOMnodes

and

target

root

nodes

of

DOM

trees

we

will

also

assume

that

only

those

DOMnodes

that

are

document

roots

can

have

an

authority

score

At

the

end

of

the

h

Ea

substep

of

MicroHITS

leaf

DOMnodes

get

a

hub

score

Because

leaf

nodes

point

to

exactly

one

page

via

an

HREF

the

hub

score

is

exactly

the

authority

score

of

the

target

page

Macroscopic

distillation

algorithms

in

effect

aggregate

all

the

leaf

hub

scores

for

a

page

into

one

hub

score

for

the

entire

page

Reporting

leaf

hub

scores

in

descending

order

would

be

useless

because

they

would

simply

follow

the

authority

ranking

and

fail

to

identify

good

hub

aggregates

Instead

of

the

total

hub

score

one

may

consider

the

density

of

hub

scores

in

a

subtree

which

may

be

defined

as

the

total

hub

score

in

the

subtree

divided

by

the

number

of

HREF

leaves

The

maximum

density

will

be

achieved

by

the

leaf

node

that

links

to

the

best

authority

In

our

experience

small

subtrees

with

small

number

of

leaves

dominate

the

top

ranks

again

under-aggregating

hub

scores

and

pitting

ancestor

scores

against

descendant

scores

A

generative

model

for

hubs

To

help

us

find

suitable

frontiers

along

which

we

can

aggregate

hub

scores

we

propose

the

following

generative

model

for

hubs

Imagine

that

the

Web

has

stopped

changing

and

with

respect

to

a

fixed

query

all

Web

pages

have

been

manually

rated

for

their

worth

as

hubs

From

these

hub

scores

one

may

estimate

that

the

hub

scores

have

been

generated

from

a

distribution

-LRB-

E.g.

may

represent

an

exponential

distribution

with

mean

-RRB-

If

the

author

of

a

hub

page

sampled

URLs

at

random

to

link

to

the

distribution

of

hub

scores

at

the

leaves

of

the

page

would

approach

the

global

distribution

provided

enough

samples

were

taken

However

authors

differ

in

their

choice

of

URLs

Hub

authors

are

not

aware

of

all

URLs

relevant

to

a

given

query

or

their

relative

authority

otherwise

all

hubs

authored

on

a

topic

would

be

complete

and

identical

and

therefore

all

but

one

would

be

pointless

to

author

-LRB-

Here

we

momentarily

ignore

the

value

added

by

annotations

and

commentaries

on

hub

pages

-RRB-

Therefore

the

distribution

of

hub

scores

for

pages

composed

by

a

specific

author

will

be

different

from

-LRB-

E.g.

the

author

's

personal

average

of

hub

scores

may

be

distributed

exponentially

-RRB-

Moreover

the

authors

of

mixed

hubs

deliberately

choose

to

dedicate

not

the

entire

page

but

only

a

fragment

or

subtree

of

it

to

URLs

that

are

relevant

to

the

given

query

-LRB-

As

an

extreme

case

a

subtree

could

be

a

single

HREF

-RRB-

We

can

regard

the

hub

generation

process

as

a

progressive

specialization

of

the

hub

score

distribution

starting

from

the

global

distribution

For

simplicity

assume

all

document

roots

are

attached

to

a

super-root

which

corresponds

to

the

global

distribution

As

the

author

works

down

the

DOMtree

corrections

are

applied

to

the

score

distribution

at

nodes

on

the

path

At

some

suitable

depth

the

author

fixes

the

score

distribution

and

generates

links

to

pages

so

that

hub

scores

follow

that

distribution

This

does

not

mean

that

there

are

no

interesting

DOMsubtrees

below

this

depth

The

model

merely

posits

that

up

to

some

depth

DOMstructure

is

indicative

of

systematic

choices

of

score

distributions

whereas

beyond

that

depth

variation

is

statistical

Global

distribution

Progressive

distortion

Model

frontier

Other

pages

v

u

Cumulative

distortion

cost

KL

-LRB-

u

-RRB-

KL

-LRB-

u

v

-RRB-

Data

encoding

cost

is

roughly

v

H

h

v

h

-RRB-

Pr

-LRB-

log

Hot

subtree

Cold

subtree

Figure

Our

fine-grained

model

of

Web

linkage

which

unifies

hyperlinks

and

DOM

structure

Discovering

DOM

frontiers

from

generated

hubs

During

topic

distillation

we

observe

pages

which

are

the

outcome

of

the

generative

process

described

above

and

our

goal

is

to

discover

the

best

frontier

at

which

the

score

distributions

were

likely

to

have

been

fixed

A

balancing

act

is

involved

here

one

may

choose

a

large

and

redundant

frontier

near

the

leaves

and

model

the

many

small

homogeneous

subtrees

-LRB-

each

with

a

different

distribution

w

-RRB-

attached

to

that

frontier

accurately

or

one

may

choose

a

short

frontier

near

the

root

with

a

few

subtrees

which

are

harder

to

model

because

they

contain

diverse

hub

scores

The

balancing

act

requires

a

common

currency

to

compare

the

cost

of

the

frontier

with

the

cost

of

modeling

hub

score

data

beneath

the

frontier

This

is

a

standard

problem

in

segmentation

clustering

and

model

estimation

A

particularly

successful

approach

to

optimizing

the

trade-off

is

to

use

the

Minimum

Description

Length

-LRB-

MDL

-RRB-

principle

-LSB-

-RSB-

MDL

provides

a

recipe

for

bringing

the

cost

of

model

corrections

to

the

same

units

as

the

cost

for

representing

data

w.r.t

a

model

and

postulates

that

learning

is

equivalent

to

minimizing

the

sum

total

of

model

and

data

encoding

costs

Data

encoding

cost

First

we

consider

the

cost

of

encoding

all

the

h-values

at

the

leaves

of

a

subtree

rooted

at

node

w.

Specifically

let

the

distribution

associated

with

w

be

w

The

set

of

HREF

leaf

nodes

in

the

subtree

rooted

at

node

w

is

denoted

L

w

and

the

set

of

hub

scores

at

these

leaves

is

denoted

H

w

As

part

of

the

solution

we

will

need

to

evaluate

the

number

of

bits

needed

to

encode

h-values

in

H

w

using

the

model

w

There

are

efficient

codes

which

can

achieve

a

data

encoding

length

close

to

Shannon

's

entropy-based

lower

bound

-LSB-

-RSB-

of

hH

w

log

Pr

w

-LRB-

h

-RRB-

bits

-LRB-

-RRB-

where

Pr

w

-LRB-

h

-RRB-

is

the

probability

of

hub

score

h

w.r.t.

a

distribution

represented

by

w

-LRB-

E.g.

w

may

include

the

mean

and

variance

of

a

normal

distribution

-RRB-

We

will

use

this

lower

bound

as

an

approximation

to

our

data

encoding

cost

-LRB-

This

would

work

if

the

h-values

followed

a

discrete

probability

distribution

which

is

not

the

case

with

hub

scores

We

will

come

back

to

this

issue

in

-RRB-

Model

encoding

cost

Next

we

consider

the

model

encoding

cost

Consider

node

v

in

the

DOMtree

We

will

assume

that

is

known

to

all

and

use

the

path

from

the

global

root

to

v

to

inductively

encode

each

node

w.r.t

its

parent

Suppose

we

want

to

specialize

the

distribution

v

of

some

v

away

from

u

the

distribution

of

its

parent

u

The

cost

for

specifying

this

change

is

given

by

the

well-known

Kullback-Leibler

-LRB-

KL

-RRB-

distance

-LSB-

-RSB-

KL

-LRB-

u

v

-RRB-

expressed

as

KL

-LRB-

u

v

-RRB-

x

Pr

u

-LRB-

x

-RRB-

log

Pr

u

-LRB-

x

-RRB-

Pr

v

-LRB-

x

-RRB-

-LRB-

-RRB-

Intuitively

this

is

the

cost

of

encoding

the

distribution

v

w.r.t.

a

reference

distribution

u

E.g.

if

X

is

a

binary

random

variable

and

its

probabilities

of

being

zero

and

one

are

-LRB-

-RRB-

under

and

-LRB-

-RRB-

under

then

KL

-LRB-

-RRB-

log

log

Unlike

in

the

case

of

entropy

the

sum

can

be

taken

to

an

integral

in

the

limit

for

a

continuous

variable

x.

Clearly

for

u

v

the

KL

distance

is

zero

it

can

also

be

shown

that

this

is

a

necessary

condition

and

that

the

KL

distance

is

asymmetric

in

general

but

always

non-negative

If

u

is

specialized

to

v

and

v

is

specialized

to

w

the

cost

is

additive

i.e.

KL

-LRB-

u

v

-RRB-

KL

-LRB-

v

w

-RRB-

We

will

denote

the

cost

of

such

a

path

as

KL

-LRB-

u

v

w

-RRB-

Moreover

the

model

encoding

cost

of

v

starting

from

the

global

root

model

will

be

denoted

KL

-LRB-

v

-RRB-

Combined

optimization

problem

Given

the

model

at

the

parent

node

u

and

the

observed

data

H

v

we

should

choose

v

so

as

to

minimize

the

sum

of

the

KL

distance

and

data

encoding

cost

KL

-LRB-

v

u

-RRB-

hH

v

log

Pr

v

-LRB-

h

-RRB-

-LRB-

-RRB-

If

v

is

expressed

parametrically

this

will

involve

an

optimization

over

those

parameters

With

the

above

set-up

we

are

looking

for

a

cut

or

frontier

F

across

the

tree

and

for

each

v

F

a

v

such

that

vF

KL

-LRB-

v

-RRB-

hH

v

log

Pr

v

-LRB-

h

-RRB-

-LRB-

-RRB-

is

minimized

The

first

part

expresses

the

total

model

encoding

cost

of

all

nodes

v

on

the

frontier

F

starting

from

the

global

root

distribution

The

second

part

corresponds

to

the

data

encoding

cost

for

the

set

of

hub

scores

H

v

at

the

leaves

of

the

subtrees

rooted

at

the

nodes

v.

Figure

illustrates

the

two

costs

Practical

considerations

The

formulation

above

is

impractical

for

a

number

of

reasons

There

is

a

reduction

from

the

knapsack

problem

to

the

frontier-finding

problem

Dynamic

programming

can

be

used

to

give

close

approximations

-LSB-

-RSB-

but

with

tens

of

thousands

of

macro-level

pages

each

with

hundreds

of

DOM

nodes

something

even

simpler

is

needed

We

describe

the

simplifications

we

had

to

make

to

control

the

complexity

of

our

algorithm

We

use

the

obvious

greedy

expansion

strategy

We

initialize

our

frontier

with

the

global

root

and

keep

picking

a

node

u

from

the

frontier

to

see

if

expanding

it

to

its

immediate

children

-LCB-

v

-RCB-

will

result

in

a

reduction

in

code

length

if

so

we

replace

u

by

its

children

and

continue

until

no

further

improvement

is

possible

We

compare

two

costs

locally

at

each

u

The

cost

of

encoding

all

the

data

in

H

u

with

respect

to

model

u

The

cost

of

expanding

u

to

its

children

i.e.

v

KL

-LRB-

u

v

-RRB-

plus

the

cost

of

encoding

the

subtrees

H

v

with

respect

to

v

If

the

latter

cost

is

less

we

expand

u

otherwise

we

prune

it

meaning

that

u

becomes

a

frontier

node

Another

issue

is

with

optimizing

the

model

v

Usually

closed

form

solutions

are

rare

and

numerical

optimization

must

be

resorted

to

again

impractical

in

our

setting

In

practice

if

H

v

is

moderately

large

the

data

encoding

cost

tends

to

be

larger

than

the

model

cost

In

such

cases

a

simple

approximation

which

works

quite

well

is

to

first

minimize

the

data

encoding

cost

for

H

v

by

picking

parameter

values

for

v

that

maximize

the

probability

of

the

observed

data

-LRB-

the

maximum

likelihood

or

ML

parameters

-RRB-

thus

fix

v

then

evaluate

KL

-LRB-

u

v

-RRB-

-LRB-

As

an

example

if

a

coin

tossed

n

times

turns

up

heads

k

times

the

ML

parameter

for

bias

is

simply

k/n

but

if

a

uniform

u

U

-LRB-

-RRB-

is

chosen

the

mean

of

v

shifts

slightly

to

-LRB-

k

-RRB-

-LRB-

n

-RRB-

which

is

a

negligible

change

for

moderately

large

k

and

n.

-RRB-

Non-parametric

evaluation

of

the

KL

distance

is

complicated

and

often

entails

density

estimates

We

experimented

with

two

parametric

distributions

the

Gaussian

and

exponential

distributions

for

which

the

KL

distance

has

closed

form

expressions

We

finally

picked

the

exponential

distribution

because

it

fit

the

observed

hub

score

distribution

more

closely

If

represents

an

exponential

distribution

with

mean

and

probability

density

f

-LRB-

x

-RRB-

-LRB-

-RRB-

exp

-LRB-

x

-RRB-

then

KL

-LRB-

-RRB-

log

-LRB-

-RRB-

where

i

corresponds

to

i

-LRB-

i

-RRB-

The

next

issue

is

how

to

measure

data

encoding

cost

for

continuous

variables

There

is

a

notion

of

the

relative

entropy

of

a

continuous

distribution

which

generalizes

discrete

entropy

but

the

relative

entropy

can

be

negative

and

is

useful

primarily

for

comparing

the

information

content

in

two

signal

sources

Therefore

we

need

to

discretize

the

hub

scores

A

common

approach

to

discretizing

real

values

is

to

scale

the

smallest

value

to

one

in

effect

allocating

log

-LRB-

h

max

h

min

-RRB-

bits

per

value

This

poses

a

problem

in

our

case

Consider

the

larger

graph

in

figure

If

h

is

initialized

to

-LRB-

-RRB-

T

after

the

first

few

multiplications

by

EE

T

which

represents

the

linear

transformation

-LRB-

h

-LRB-

-RRB-

h

-LRB-

-RRB-

-RRB-

T

-LRB-

h

-LRB-

-RRB-

h

-LRB-

-RRB-

h

-LRB-

-RRB-

-LRB-

-RRB-

h

-LRB-

-RRB-

-RRB-

T

we

get

-LRB-

-RRB-

T

-LRB-

-RRB-

T

-LRB-

-RRB-

T

and

-LRB-

-RRB-

T

Even

if

we

disregard

the

zeroes

the

ratio

of

the

largest

to

the

smallest

positive

component

of

h

grows

without

bound

As

scaling

is

employed

to

prevent

overflow

h

-LRB-

-RRB-

decays

towards

zero

This

makes

the

log

-LRB-

h

max

h

min

-RRB-

strategy

useless

A

reasonable

compromise

is

possible

by

noting

that

the

user

is

not

interested

in

the

precision

of

all

the

hub

scores

E.g.

reporting

the

top

fraction

of

positive

hub

scores

to

within

a

small

multiplicative

error

of

is

quite

enough

We

used

and

in

our

experiments

Distillation

using

segmented

hubs

In

this

section

we

will

embed

the

segmentation

algorithm

discussed

in

the

previous

section

into

the

edge-weighted

B&H

algorithm

-LRB-

Unlike

the

full

B&H

algorithm

we

do

no

text

analysis

at

this

stage

We

continue

to

call

the

edge-weighted

version

of

HITS

as

B&H

for

simplicity

-RRB-

The

main

modification

will

be

the

insertion

of

a

call

to

the

segmentation

algorithm

after

the

h

Ea

step

and

before

the

complementary

step

a

E

T

h

It

is

also

a

reasonable

assumption

that

the

best

frontier

will

segment

each

hub

non-trivially

i.e.

below

its

DOMroot

Therefore

we

can

invoke

the

segmentation

routine

separately

on

each

page

Let

the

segmentation

algorithm

described

previously

be

invoked

as

F

segment

-LRB-

u

-RRB-

where

u

is

the

DOMtree

root

of

a

page

and

F

is

the

returned

frontier

for

that

page

Here

is

the

pseudo-code

for

one

iteration

h

Ea

for

each

document

DOMroot

u

F

segment

-LRB-

u

-RRB-

for

each

frontier

node

v

F

h

-LRB-

v

-RRB-

wL

v

h

-LRB-

w

-RRB-

for

each

w

L

v

h

-LRB-

w

-RRB-

h

-LRB-

v

-RRB-

reset

h

-LRB-

v

-RRB-

a

E

T

h

normalize

a

so

that

u

a

-LRB-

u

-RRB-

For

convenience

we

can

skip

the

hub

normalization

and

only

normalize

authorities

every

complete

cycle

this

does

not

affect

ranking

The

reader

will

observe

that

this

is

not

a

linear

relaxation

as

was

the

case

with

HITS

Clever

or

B&H

because

segment

may

lead

us

to

aggregate

and

redistribute

different

sets

of

hub

scores

in

different

iterations

based

on

the

current

leaf

hub

scores

-LRB-

Also

note

that

if

F

were

fixed

for

each

page

for

all

time

the

system

would

still

be

linear

and

therefore

guaranteed

to

converge

-RRB-

Although

convergence

results

for

non-linear

dynamical

systems

are

rare

-LSB-

-RSB-

in

our

experiments

we

never

found

convergence

to

be

a

problem

-LRB-

see

-RRB-

However

we

do

have

to

take

care

with

the

initial

values

of

a

and

h

unlike

in

the

linear

relaxation

situation

where

any

positive

value

will

do

Assume

that

the

first

iteration

step

transfers

weights

from

authorities

to

hubs

and

consider

how

we

can

initialize

the

authority

scores

In

contrast

to

HITS

we

can

not

start

with

all

a

-LRB-

v

-RRB-

Why

not

Because

both

good

and

bad

authorities

will

get

this

score

resulting

in

many

hub

DOMsubtrees

looking

more

uniformly

promising

than

they

should

This

will

lead

the

segment

algorithm

to

prune

the

frontier

too

eagerly

resulting

in

potentially

excessive

authority

diffusion

as

in

HITS

We

propose

a

more

conservative

initialization

policy

Similar

to

B&H

we

assume

that

the

textual

content

of

the

rootset

documents

returned

by

the

text

search

engine

is

more

reliably

relevant

than

the

neighbors

included

for

distillation

Therefore

we

start

our

algorithm

by

assigning

only

root-set

authority

scores

to

one

Of

course

once

the

iterations

start

this

does

not

prevent

authority

from

diffusing

over

to

siblings

but

the

diffusion

is

controlled

by

hub

segmentation

There

is

one

other

way

in

which

we

bias

our

algorithm

to

be

conservative

w.r.t.

authority

diffusion

If

a

DOMnode

has

only

one

child

with

a

positive

hub

score

or

if

there

is

a

tie

in

the

cost

of

expanding

vs.

pruning

we

expand

the

node

thereby

pushing

the

frontier

down

and

preventing

the

leaf

hub

score

from

spreading

out

to

possibly

irrelevant

outlinks

Taken

together

these

two

policies

may

be

a

little

too

conservative

sometimes

preventing

desirable

authority

diffusion

and

bringing

our

algorithm

closer

to

MicroHITS

than

we

would

like

For

example

the

graph

being

distilled

may

be

such

that

page

u

has

one

DOMsubtree

clearly

-LRB-

to

a

human

reading

the

text

-RRB-

dedicated

to

motorcycles

but

only

one

link

target

v

is

in

the

expanded

set

In

ongoing

work

we

are

integrating

text

analysis

into

our

fine-grained

model

to

avoid

such

pitfalls

-LSB-

-RSB-

We

used

the

queries

used

in

the

Clever

studies

-LSB-

-RSB-

and

by

B&H

-LSB-

-RSB-

-LRB-

shown

in

Figure

-RRB-

For

each

RagingSearch

returned

at

most

responses

in

the

root

set

These

pages

were

fetched

and

all

their

outlinks

included

in

our

database

as

well

RagingSearch

and

HotBot

were

used

to

get

as

many

inlinks

to

the

root

set

as

possible

these

were

also

included

in

our

database

This

resulted

in

about

raw

URLs

After

normalizing

URLs

and

eliminating

duplicates

ap-proximately

page

fetches

succeeded

We

used

the

command-line

page

fetching

tool

from

org

for

its

reliable

timeout

mechanism

We

then

scanned

all

these

pages

and

filled

a

global

-LRB-

macro

-RRB-

link

table

with

non-local

links

i.e.

links

between

pages

not

on

the

same

hostname

-LRB-

as

a

lowercase

string

without

port

number

-RRB-

We

then

proceeded

to

parse

the

documents

into

their

DOMs

in

order

to

populate

a

different

set

of

tables

that

represented

the

DOMnodes

and

the

micro-links

between

them

We

used

the

javax.swing.text.html.parser

package

and

built

a

custom

pared-down

DOMgenerator

on

top

of

the

SAX

scanner

provided

The

total

number

of

micro-links

was

and

the

total

number

of

micro-nodes

likewise

increased

Out

of

the

two

million

non-local

links

less

than

had

targets

that

were

not

the

root

of

the

DOMtree

of

a

page

Thus

our

introduction

of

the

asymmetry

in

handling

hubs

and

authorities

seems

to

be

not

a

great

distortion

of

reality

Even

though

our

experiments

were

performed

on

a

MHz

Pentium

Xeon

processor

with

MB

RAM

and

GB

of

disk

handling

this

scale

of

operation

required

some

care

and

compromise

In

particular

to

cut

down

the

micro-graph

to

only

about

million

edges

we

deleted

all

DOMpaths

that

did

not

lead

to

an

A.

A

element

Otherwise

we

estimated

that

the

number

of

micro-links

would

be

at

least

two

orders

of

magnitude

larger

In

our

ongoing

work

we

are

having

to

address

this

issue

as

we

are

also

analyzing

text

Query

Drift

Mixed

affirmative

action

large

alcoholism

amusement

park

small

architecture

bicycling

blues

classical

guitar

small

cheese

cruises

computer

vision

field

hockey

gardening

graphic

design

large

Gulf

war

large

HIV

lyme

disease

small

mutual

fund

small

parallel

architecture

rock

climbing

large

recycling

can

stamp

collecting

Shakespeare

sushi

small

telecommuting

large

Thailand

tourism

large

table

tennis

small

vintage

cars

small

Zen

buddhism

large

Figure

The

set

of

broad

queries

used

for

comparing

B&H

-LRB-

without

text

analysis

-RRB-

and

our

system.The

second

column

shows

the

extent

of

drift

in

the

B&H

response.The

third

column

shows

if

mixed

hubs

were

found

within

the

top

hubs

reported

Figure

shows

the

queries

used

by

the

Clever

study

and

by

B&H

As

indicated

before

our

baseline

was

B&H

with

edge-weighting

but

without

text-based

outlier

elimination

which

we

will

simply

call

B&H

We

did

not

have

any

arbitrary

cut-off

for

the

number

of

in-links

used

as

we

did

not

know

which

to

discard

and

which

to

keep

As

B&H

noted

edge-weighting

improved

results

significantly

but

without

text

analysis

is

not

adequate

to

prevent

drift

Of

the

queries

half

show

drift

to

some

extent

We

discuss

a

few

cases

Affirmative

action

is

understandably

dominated

by

lists

of

US

universities

because

they

publicize

their

support

for

the

same

Less

intuitive

was

the

drift

into

the

world

of

software

until

we

found

html

in

the

root

set

which

presents

a

dialup

networking

software

called

Affirmative

Action

and

links

to

many

popular

freeware

sites

-LRB-

figure

-RRB-

By

itself

this

page

would

not

survive

the

link-based

ranking

but

the

clique

of

software

sites

leads

B&H

astray

Another

example

was

amusement

parks

where

B&H

fell

prey

to

multi-host

nepotism

in

spite

of

edge-weighting

A

densely

connected

conglomerate

including

the

relevant

starting

point

-LRB-

figure

-RRB-

formed

a

multi-site

nepotistic

cluster

and

misled

macroscopic

algorithms

In

both

these

cases

there

were

ample

clues

in

the

DOM

structure

alone

-LRB-

leave

alone

text

-RRB-

that

authority

diffusion

should

be

suppressed

We

obtained

several

cases

of

reduced

drift

using

our

technique

-LRB-

In

ongoing

work

we

are

getting

the

improvement

evaluated

by

volunteers

-RRB-

One

striking

example

was

for

the

query

amusement

parks

where

our

algorithm

prevented

from

taking

over

the

show

-LRB-

see

figure

complete

results

are

in

AP-macro

html

and

AP-micro.html

-RRB-

Figure

The

part

of

this

HTML

page

that

contains

the

query

affirmative

action

is

not

very

popular

but

adjoining

DOM

subtrees

-LRB-

upper

right

corner

-RRB-

create

a

dense

network

of

software

sites

and

mislead

macroscopic

distillation

algorithms.Dotted

red

lines

are

drawn

by

hand

Figure

The

clique

attack

comprises

a

set

of

sibling

sites

with

different

hostnames

and

a

wide

variety

of

topics

linking

to

each

other.A

human

can

easily

avoid

paying

attention

to

the

sibling

sites

but

macroscopic

distillation

will

get

misled.Dotted

red

lines

are

drawn

by

hand

Figure

also

shows

that

for

almost

half

the

queries

we

found

excellent

examples

of

mixed

hubs

within

the

top

hubs

reported

Given

the

abundance

of

hubs

on

these

topics

we

had

anticipated

that

the

best

hubs

would

be

pure

While

this

was

to

some

extent

true

we

found

quite

a

few

mixed

hubs

too

Our

system

automatically

highlighted

the

most

relevant

DOMsubtree

we

present

some

examples

in

figure

and

urge

the

reader

to

sample

the

annotated

hubs

packaged

with

the

HTML

version

of

this

paper

Macroscopic

Fine-grained

Figure

The

fine-grained

algorithm

is

less

susceptible

to

clique

attacks.The

query

here

is

amusement

parks

Figure

Two

samples

of

mixed

hub

annotations

amusement

parks

amidst

roller-coaster

manufacturers

and

sushi

amidst

international

cuisine

Query

Annotated

file

alcoholism

Amusement

parks

Architecture

Classical

guitar

HIV

Shakespeare

Sushi

We

verified

that

our

smoothing

algorithm

was

performing

non-trivial

work

it

was

not

merely

locating

top-scoring

authorities

and

highlighting

them

Within

the

highlighted

regions

we

typically

found

as

many

unvisited

links

as

links

already

rated

as

authorities

In

ongoing

work

we

are

using

these

new

links

for

enhanced

focused

crawling

A

key

concern

for

us

was

whether

the

smoothing

iterations

will

converge

or

not

Because

the

sites

of

hub

aggregation

are

data-dependent

the

transform

was

non-linear

and

we

could

not

give

a

proof

of

convergence

In

practice

we

faced

no

problems

with

convergence

figure

is

typical

of

all

queries

This

raised

another

concern

was

the

smoothing

subroutine

doing

anything

dynamic

and

useful

or

was

convergence

due

to

its

picking

the

same

sites

for

hub

aggregation

every

time

In

figures

and

we

plot

relative

numbers

of

nodes

pruned

vs.

expanded

against

the

number

of

iterations

Queries

which

do

not

have

a

tendency

to

drift

look

like

figure

Initially

both

numbers

are

small

As

the

system

bootstraps

into

controlled

authority

diffusion

more

candidate

hubs

are

pruned

i.e.

accepted

in

their

entirety

Diffused

authority

scores

in

turn

lead

to

fewer

nodes

Iterations

M

e

an

aut

h

s

c

o

r

e

c

h

ang

e

Figure

In

spite

of

the

non-linear

nature

of

our

relaxation

algorithm

convergence

is

quick

in

practice.A

typical

chart

of

average

change

to

authority

scores

is

shown

against

successive

iterations

ting

expanded

For

queries

with

a

strong

tendency

to

drift

-LRB-

figure

-RRB-

the

number

of

nodes

expanded

does

not

drop

as

low

as

in

low-drift

situations

For

all

the

queries

the

respective

counts

stabilize

within

iterations

#Prune

#Expand

Data

Figure

Our

micro-hub

smoothing

technique

is

highly

adaptive

the

number

of

nodes

pruned

vs.expanded

changes

dramatically

across

iterations

but

stabilizes

within

iterations

There

is

also

a

controlled

induction

of

new

nodes

into

the

response

set

owing

to

authority

diffusion

via

relevant

DOM

subtrees

-LRB-

query

bicycling

-RRB-

#Prune

#Expand

Data

Figure

For

some

queries

for

which

B&H

showed

high

drift

our

algorithm

continues

to

expand

a

relatively

larger

number

of

nodes

in

an

attempt

to

suppress

drift

-LRB-

query

affirmative

action

-RRB-

Finally

we

checked

how

close

we

were

to

B&H

ranking

We

expected

our

ranking

to

be

correlated

with

theirs

but

verified

that

there

are

meaningful

exceptions

Figure

show

a

scatter

plot

of

authority

scores

It

illustrates

that

we

systematically

under-rate

authorities

compared

to

B&H

-LRB-

the

axes

have

incomparable

scale

the

leading

straight

line

should

be

interpreted

as

y

x

-RRB-

This

is

a

natural

outcome

of

eliminating

pseudo-authorities

that

gain

prominence

in

B&H

via

mixed

hubs

Authority

score

B&H

O

u

r

a

ut

h

o

ri

t

y

s

c

ore

Figure

Our

ranking

is

correlated

to

B&H

but

not

identical

we

tend

to

systematically

under-rate

authorities

compared

to

B&H

We

have

presented

a

fine-grained

approach

to

topic

distillation

that

integrates

document

substructure

-LRB-

in

the

form

of

the

Document

Object

Model

-RRB-

with

regular

hyperlinks

Plugging

in

the

fine-grained

graph

in

place

of

the

usual

coarse-grained

graph

does

not

work

because

the

fine-grained

graph

may

not

have

the

bipartite

cores

so

vital

to

the

success

of

macroscopic

distillation

algorithms

We

propose

a

new

technique

for

aggregating

and

propagating

micro-hub

scores

at

a

level

determined

by

the

Minimum

Description

Length

principle

applied

to

the

DOMtree

with

hub

scores

at

the

leaves

We

show

that

the

resulting

procedure

still

converges

in

practice

reduces

drift

and

is

moreover

capable

of

identifying

and

extracting

regions

-LRB-

DOMsubtrees

-RRB-

relevant

to

the

query

out

of

a

broader

hub

or

a

hub

with

additional

less-relevant

contents

and

links

In

ongoing

work

apart

from

completing

a

detailed

user

study

-LRB-

as

in

the

Clever

project

-RRB-

we

are

exploring

three

more

ideas

First

our

algorithm

depends

on

DOMbranch

points

to

be

able

to

separate

relevant

hub

fragments

from

irrelevant

ones

We

have

seen

some

pages

with

a

long

sequence

of

URLs

without

any

helpful

DOMstructure

such

as

LI

providing

natural

segment

candidates

Second

we

need

to

bring

back

some

of

the

text

analysis

techniques

that

have

improved

HITS

and

integrate

them

with

our

model

Third

we

are

measuring

if

the

link

localization

done

by

our

system

can

help

in

faster

resource

discovery

Acknowledgment

Thanks

to

Vivek

Tawde

and

Hrishikesh

Gupta

for

helpful

discussions

to

S.

Sudarshan

for

stimulating

discussions

and

generous

support

from

the

Informatics

Lab

IIT

Bombay

and

the

anonymous

reviewers

for

helping

to

improve

the

presentation

-LSB-

-RSB-

E.Amitay

and

C.Paris

Automatically

summarising

web

sites

Is

there

a

way

around

it

In

International

Conference

on

Information

and

Knowledge

Management

-LRB-

CIKM

-RRB-

Washington

DC

USA

ACM

Online

at

-LSB-

-RSB-

K.Bharat

and

M.Henzinger

Improved

algorithms

for

topic

distillation

in

a

hyperlinked

environment.In

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

pages

Aug.

Online

at

-LSB-

-RSB-

S.Brin

and

L.Page

The

anatomy

of

a

large-scale

hypertextual

web

search

engine.In

Proceedings

of

the

World-Wide

Web

Conference

-LRB-

-RRB-

Online

at

-LSB-

-RSB-

O.Buyukkokten

H.Garcia-Molina

and

A.Paepcke

Focused

web

searching

with

PDAs.In

World

Wide

Web

Conference

Amsterdam

May

Online

at

-LSB-

-RSB-

S.Chakrabarti

B.Dom

D.Gibson

J.Kleinberg

P.Raghavan

and

S.Rajagopalan

Automatic

resource

compilation

by

analyzing

hyperlink

structure

and

associated

text.In

World-wide

web

conference

-LRB-

-RRB-

Online

at

-LSB-

-RSB-

S.Chakrabarti

B.E.Dom

S.Ravi

Kumar

P.Raghavan

S.Rajagopalan

A.Tomkins

D.Gibson

and

J.Kleinberg

Mining

the

Web

's

link

structure

IEEE

Computer

-LRB-

-RRB-

Aug.

-LSB-

-RSB-

S.Chakrabarti

M.Joshi

and

V.Tawde

Enhanced

topic

distillation

using

text

markup

tags

and

hyperlinks

Submitted

for

publication

Jan.

-LSB-

-RSB-

S.Chakrabarti

M.van

den

Berg

and

B.Dom

Focused

crawling

a

new

approach

to

topic-specific

web

resource

discovery

Computer

Networks

First

appeared

in

the

International

World

Wide

Web

Conference

Toronto

May

Available

online

at

-LSB-

-RSB-

T.M.Cover

and

J.A.Thomas

Elements

of

Information

Theory.John

Wiley

and

Sons

Inc.

-LSB-

-RSB-

D.A.Gibson

J.M.Kleinberg

and

P.Raghavan.Clustering

categorical

data

An

approach

based

on

dynamical

systems

In

VLDB

volume

pages

New

York

Aug.

-LSB-

-RSB-

G.H.Golub

and

C.F.van

Loan

Matrix

Computations

Johns

Hopkins

University

Press

London

-LSB-

-RSB-

M.Hearst

Multi-paragraph

segmentation

of

expository

text.In

Proceedings

of

the

Annual

Meeting

of

the

Association

for

Computational

Linguistics

Las

Cruces

NM

June

Online

at

hearst/publications

shtml

-LSB-

-RSB-

D.S.Johnson

and

K.A.Niemi

On

knapsacks

partitions

and

a

new

dynamic

programming

technique

for

trees

Mathematics

of

Operations

Research

-LRB-

-RRB-

-LSB-

-RSB-

J.Kleinberg

Authoritative

sources

in

a

hyperlinked

environment.In

ACM-SIAM

Symposium

on

Discrete

Algorithms

Online

at

-LSB-

-RSB-

R.Lempel

and

S.Moran

The

stochastic

approach

for

link-structure

analysis

-LRB-

SALSA

-RRB-

and

the

TKC

effect.In

pages

Amsterdam

May

Online

at

http

-LSB-

-RSB-

J.Rissanen

Stochastic

complexity

in

statistical

inquiry

In

World

Scientific

Series

in

Computer

Science

volume

World

Scientific

Singapore

-LSB-

-RSB-

G.Salton

and

M.J.McGill

Introduction

to

Modern

Information

Retrieval.McGraw-Hill

-LSB-

-RSB-

S.Sarawagi

Explaining

differences

in

multidimensional

aggregates.In

International

Conference

on

Very

Large

Databases

-LRB-

VLDB

-RRB-

volume

Online

at

http

dom
hyperlink
hits
text analysis
google
segmentation
document object model
pagerank algorithm
minimum description length principle
xml
link localization
topic distillation

microscopic distillation
hub fragmentation