Measuring

e-Government

Impact

Existing

practices

and

shortcomings

Public

administrations

of

all

over

the

world

invest

an

enormous

amount

of

resources

in

e-government

How

the

success

of

e-government

can

be

measured

is

often

not

clear

E-government

involves

many

aspects

of

public

administration

ranging

from

introducing

new

technology

to

business

process

-LRB-

re

-RRB-

engineering

The

measurement

of

the

effectiveness

of

e-government

is

a

complicated

endeavor

In

this

paper

current

practices

of

e-government

measurement

are

evaluated

A

number

of

limitations

of

current

measurement

instruments

are

identified

Measurement

focuses

predominantly

on

the

front

-LRB-

primarily

counting

the

number

of

services

offered

-RRB-

and

not

on

the

back-office

processes

Interpretation

of

measures

is

difficult

as

all

existing

measurement

instruments

lack

a

framework

depicting

the

relationships

between

the

indicators

and

the

use

of

resources

The

different

measures

may

fit

the

aim

of

the

owners

of

the

e-governmental

services

however

due

to

conflicting

aims

and

priorities

little

agreement

exists

on

a

uniform

set

of

measures

needed

for

comparison

of

e-government

development

Traditional

methods

of

measuring

e-government

impact

and

resource

usage

fall

short

of

the

richness

of

data

required

for

the

effective

evaluation

of

e-government

strategies

J.

-LSB-

Administrative

Data

Processing

-RSB-

Administrative

Data

Processing

Government

K.

-LSB-

Computing

Milieux

-RSB-

Computers

and

Society

Electronic

Commerce

Management

Measurement

Performance

Standardization

Public

institutions

as

well

as

business

organizations

use

the

Internet

to

deliver

a

wide

range

of

information

and

services

at

an

increasing

level

of

sophistication

-LSB-

-RSB-

However

Web

sites

and

related

business

processes

and

information

systems

are

so

complex

that

it

is

difficult

for

governments

to

determine

adequate

measures

for

evaluating

the

efficiency

and

effectiveness

of

the

spending

of

their

public

money

Moreover

only

measuring

the

front

of

public

websites

is

a

too

narrow

view

on

e-government

E-government

involves

the

collaboration

and

communication

between

stakeholders

and

integration

of

cross-agency

business

processes

An

important

aim

of

having

a

well-funded

theory

on

measuring

e-government

is

to

allow

comparison

or

benchmarking

By

examining

the

results

of

such

benchmarks

we

might

be

able

to

distinct

good

from

bad

practices

and

to

give

directives

to

designers

of

e-governmental

services

Moreover

is

should

help

to

identify

how

effective

public

money

is

spend

Thus

evaluating

the

relationship

between

results

and

resources

used

Comparison

can

have

different

goals

Selecting

between

options

is

just

one

of

them

Principally

we

can

distinct

three

types

of

comparison

-RRB-

comparison

of

alternatives

-LRB-

e.g.

to

make

a

choice

between

alternative

solutions

-RRB-

-RRB-

vertical

comparison

over

time

-LRB-

e.g.

in

order

to

measure

improvement

of

versions

-RRB-

and

-RRB-

horizontal

comparison

-LRB-

e.g.

benchmarking

different

solutions

-RRB-

Whatever

type

of

comparison

we

choose

we

can

only

compare

if

we

have

a

set

of

preferred

outcomes

Many

measurement

instruments

currently

applied

are

not

described

in

such

way

that

the

preferences

underlying

the

instrument

have

been

made

explicitly

In

this

research

we

describe

different

measurement

instruments

used

and

we

will

discuss

their

strengths

and

weaknesses

The

goal

of

this

paper

is

to

evaluate

how

current

measurement

instruments

the

impact

of

e-government

The

combined

research

methodology

of

literature

research

and

case

study

was

chosen

to

answer

the

goal

of

this

research

Case

study

research

can

be

characterized

as

qualitative

and

observatory

using

predefined

research

questions

-LSB-

-RSB-

Case

studies

were

used

to

examine

existing

measurement

instruments

in

the

Netherlands

The

e-government

monitor

of

Accenture

and

a

European

Union

measurement

instrument

was

also

evaluated

as

thes

instruments

are

used

by

Dutch

agencies

to

benchmark

their

situation

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

or

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

ICEC

Sixth

International

Conference

on

Electronic

Commerce

Edited

by

Marijn

Janssen

Henk

G.

Sol

and

Ren

W.

Wagenaar

Copyright

ACM

Before

analyzing

instruments

developed

to

measure

e-government

development

it

is

necessary

to

understand

the

nature

of

e-government

The

organization

of

public

administration

involves

many

heterogeneous

types

of

business

processes

When

law

or

regulations

are

changed

these

processes

and

their

supportive

systems

have

to

be

adapted

Within

one

policy

layer

the

process

of

adaptation

starts

with

legislation

drafting

-LRB-

adapting

the

law

or

regulations

-RRB-

followed

by

a

chain

of

processes

varying

from

translating

these

law

texts

into

specifications

design

of

processes

and

supporting

systems

development

of

these

processes

and

systems

and

finally

implementation

and

use

-LRB-

for

a

recent

legal

engineering

approach

see

-LSB-

-RSB-

-RRB-

A

complicating

factor

is

that

more

than

one

governmental

layer

exists

and

often

interaction

between

these

layers

occurs

Especially

the

need

to

adapt

legislation

from

the

European

Union

is

a

more

then

ever

dominant

factor

that

even

further

complicates

this

process

is

In

Figure

the

fragmented

nature

of

the

government

is

shown

Legislation

and

service

provisioning

efforts

are

distributed

over

the

European

State

Region

and

local

level

The

situation

is

even

more

complicated

as

within

each

level

many

agencies

of

various

types

exists

At

a

local

level

municipalities

water

boards

chambers

of

commerce

local

taxes

and

many

other

public

and

public-private

agencies

exists

As

such

many

agencies

influence

the

impact

of

e-government

and

measurement

should

include

them

all

Europe

Europe

Europe

Europe

Europe

state

Europe

region

Europe

Europe

local

int

e

r

oper

ata

b

ility

int

e

roper

a

t

ability

int

e

r

opera

t

a

bility

policy

and

standardization

navigation

service

provisioning

helpdesk

and

appeal

Aggregation

of

local

initaitves

navigagtion

geogprahical

relevance

standardization

policies

culture

enforcement

businesses

demand

Figure

Fragmented

nature

of

public

administration

Two

main

types

of

interactions

influence

this

fragmented

landscape

the

policy-making

implementation

execution

and

enforcement

of

new

legislation

and

the

business

and

citizens

searching

for

information

and

services

We

will

denote

the

creation

implementation

execution

enforcement

and

maintenance

of

laws

as

production

cycle

in

this

paper

Governments

are

looking

for

ways

to

increase

their

efficiency

effectiveness

decrease

the

administrative

burden

and

reduce

the

lead

times

for

adopting

new

legislations

The

consequences

of

new

laws

at

production

phase

-LRB-

drafting

-RRB-

are

only

roughly

known

Only

after

implementing

the

new

regulations

in

the

processes

and

supporting

systems

the

full

meaning

of

applying

these

regulations

becomes

clear

Certainly

when

the

interpretations

and

translation

into

practical

applications

take

place

at

local

government

level

it

often

will

not

be

possible

to

timely

inform

the

businesses

or

citizens

who

are

affected

by

the

new

law

pro-actively

as

no

information

about

concrete

effects

on

their

cases

is

available

A

complicating

factor

is

that

many

times

a

administrative

process

is

supported

by

heterogeneous

information

systems

and

many

different

parties

can

be

involved

in

adapting

those

systems

Most

of

the

companies

ERP

software

components

need

to

be

updated

quite

frequently

to

be

in

and

keep

being

in

accordance

with

small

changes

in

administrative

legislation

The

same

holds

for

Human

Resources

software

Time

reporting

software

Tax

reporting

software

and

more

indirectly

Logistics

software

All

have

to

be

updated

due

to

changes

in

legislation

It

does

not

require

extensive

explanation

to

stress

the

need

for

smart

public-private

networks

from

a

production

chain

perspective

Nowadays

businesses

expect

that

the

governments

reduce

the

administrative

burden

for

businesses

Governments

can

achieve

this

goal

by

creating

a

smart

service

oriented

public

administration

To

be

able

to

provide

these

integrated

services

access

to

different

legal

sources

or

better

to

the

formal

models

that

could

be

derived

from

those

models

is

needed

-LRB-

see

e.g.

-LSB-

-RSB-

-RRB-

Standardization

initiatives

like

the

Metalex

standard

for

describing

legal

sources

-LRB-

see

-RRB-

and

the

Power

method

for

modeling

normative

systems

-LRB-

see

-RRB-

are

essential

first

steps

towards

this

They

provide

a

basis

for

interoperable

and

contextual

better

understandable

and

accessible

legal

sources

that

could

easier

be

connected

to

the

context

of

business

activities

From

the

demand

perspective

citizens

and

businesses

find

it

very

hard

to

access

relevant

legislation

local

procedures

and

rules

policy

documents

etc.

Governmental

bodies

are

engaged

in

a

flurry

of

policy

and

law

making

activities

Not

only

is

this

a

complex

myriad

of

legal

issues

but

the

information

is

produced

at

different

levels

of

public

administration

including

local

regional

national

and

European

union

A

commonly

accepted

requirement

however

is

that

online

state

legislative

information

is

equally

accessible

to

all

-LRB-

Fage

Fagan

-RRB-

and

of

course

in

the

first

place

it

should

be

accessible

Many

governments

currently

are

searching

for

ways

to

make

their

information

accessible

and

retrievable

This

involves

issues

regarding

terminology

explaining

the

type

of

legislative

document

understandable

and

easy-to-use

search

interfaces

and

accessing

the

official

status

of

online

documents

A

central

question

for

researchers

working

in

the

field

of

e-government

is

how

to

measure

the

progress

made

in

complying

to

the

requirements

mentioned

before

In

this

paper

we

examine

some

examples

of

measurement

instruments

that

were

developed

to

measure

progress

in

e-Government

But

before

describing

these

instruments

we

will

discuss

some

literature

on

measuring

eGovernment

first

There

is

a

diversity

of

literature

focusing

on

measurements

Stage

models

are

often

used

to

positioning

and

evaluate

the

current

status

of

e-government

development

Services

literature

focusing

on

the

measurement

to

of

perceived

quality

satisfaction

of

complex

multi-service

organizations

Last

there

is

the

research

focusing

on

developing

suitable

yardstick

performance

indicators

acting

as

surrogates

for

measuring

the

performance

Stage

models

Many

researchers

on

e-business

address

the

stages

of

Web

site

development

Green

-LSB-

-RSB-

suggests

three

stages

attracting

transforming

and

utilization

of

media

technology

Fink

et

al.

-LRB-

-RRB-

focuses

on

attracting

enhancing

and

retaining

client

relationships

using

the

Web

site

applications

Moon

-LSB-

-RSB-

proposes

a

five

stage

model

Layne

and

Lee

-LSB-

-RSB-

propose

four

stages

of

a

growth

model

towards

e-government

In

stage

one

governments

create

a

state

website

by

establishing

a

separate

Internet

department

There

is

no

integration

between

the

processes

in

the

front

and

back

offices

The

web

sites

are

focused

on

creating

a

web-presence

and

providing

information

Transaction

possibilities

for

customers

are

restricted

to

printing

online

forms

and

sending

them

by

traditional

mail

At

stage

two

transaction

there

is

two-way

communication

Customers

transact

with

government

on-line

by

filling

out

forms

and

government

responds

by

providing

confirmations

receipts

etc.

The

number

of

transactions

is

relatively

small

and

the

organization

becomes

confronted

with

translating

information

from

and

back

to

the

front

office

and

the

other

way

around

Often

a

working

database

is

established

in

the

front

office

for

supporting

immediate

transactions

The

data

in

this

database

is

periodically

derived

from

and

exported

to

the

various

databases

in

the

back

office

At

stage

three

vertical

integration

the

focus

is

moving

towards

transformation

of

government

services

rather

than

automating

and

digitizing

existing

processes

Most

information

systems

are

localized

and

fragmented

Government

organizations

often

maintain

separate

databases

that

are

not

connected

to

other

governmental

agencies

at

the

same

level

or

with

similar

agencies

at

the

local

or

federal

level

A

natural

progression

according

to

Layne

and

Lee

is

the

integration

of

scattered

information

systems

at

different

levels

-LRB-

vertical

-RRB-

of

government

services

Layne

and

Lee

-LRB-

-RRB-

expect

that

vertical

integration

within

the

similar

functional

walls

but

across

different

levels

of

government

will

happen

first

because

the

gap

between

levels

of

government

is

much

less

than

the

difference

between

different

functions

Information

systems

are

connected

to

each

other

or

at

least

can

communicate

with

each

other

The

problem

with

the

description

of

governmental

systems

is

that

they

do

n't

make

a

distinction

between

the

-LRB-

legal

and

administrative

knowledge

contained

in

those

systems

and

the

data

-LRB-

of

citizens

-RRB-

to

which

that

knowledge

is

applied

-LRB-

e.g.

to

derive

if

someone

is

entitled

to

receive

a

subsidy

-RRB-

Especially

the

sharing

of

data

is

limited

and

for

very

good

reasons

too

Both

the

desire

to

guarantee

a

certain

level

of

privacy

and

the

vulnerability

for

misuse

of

data

have

been

reasons

for

the

legislator

to

limit

storage

reusing

and

sharing

data

between

different

agencies

-LRB-

not

to

speak

of

passing

data

of

citizens

from

the

government

to

private

institutions

-RRB-

The

challenge

consequently

is

how

to

realize

the

full

potential

of

information

technology

from

the

customer

's

perspective

since

this

can

only

be

achieved

by

horizontally

integrating

government

services

across

different

functional

walls

-LRB-

silos

-RRB-

in

Layne

and

Lee

's

stage

four

horizontal

integration

The

question

is

how

to

achieve

this

without

the

need

for

having

databases

across

different

functional

areas

to

communicate

with

each

The

situation

that

information

obtained

by

one

agency

can

be

used

for

CRM

purposes

by

other

agencies

by

sharing

information

is

for

reasons

mentioned

earlier

undesirable

The

knowledge

contained

in

these

information

systems

however

can

be

shared

and

reused

thus

allowing

better

integrated

government

services

Service

literature

The

concepts

of

perceived

quality

and

satisfaction

are

two

of

the

fundamental

pillars

of

recent

evaluation

studies

Bign

et

al.

-LSB-

-RSB-

conclude

that

in

most

cases

the

fundamental

unit

of

analysis

has

been

an

isolated

service

and

the

fact

that

several

services

may

be

offered

by

an

individual

organization

has

not

been

taken

into

account

Indeed

multi-service

organizations

where

the

customer

has

access

to

several

services

have

not

been

so

intensively

dealt

with

The

problems

facing

these

organizations

in

terms

of

measurement

and

management

of

perceived

quality

and

of

satisfaction

are

more

complex

than

in

those

organizations

where

only

one

service

is

offered

but

less

complex

then

situations

where

a

service

has

to

be

assembled

from

many

other

service

suppliers

When

measuring

the

quality

of

such

integrated

service

it

is

necessary

to

take

into

consideration

not

only

the

perceived

quality

of

each

of

the

elementary

services

but

also

the

perceived

overall

quality

of

the

constituting

ones

Bign

et

al.

-LSB-

-RSB-

found

that

the

scale

used

to

determine

the

perceived

quality

of

the

core

services

of

hospitals

and

universities

is

composed

of

five

dimensions

as

proposed

by

Parasuraman

et

al.

-LSB-

-RSB-

tangibility

reliability

responsiveness

confidence

and

empathy

Performance

indicators

Performance

indicators

serve

as

surrogates

to

deduce

the

quality

of

the

e-government

performance

-LSB-

-RSB-

Lee

-LSB-

-RSB-

provides

measurements

based

on

development

states

and

a

modification

of

Simeon

's

-LSB-

-RSB-

components

-LRB-

-RRB-

The

following

five

items

determine

the

affect

-LRB-

Attracting

-RRB-

of

a

homepage

on

the

Web

site

Design

of

logo

and

tagline

-LRB-

quick

summary

of

what

a

Web

site

is

all

about

-RRB-

Graphics

-LRB-

e.g.

layout

color

and

figures

of

a

homepage

-RRB-

Institution

's

self-advertising

-LRB-

e.g.

banner

button

and

interstitials

-RRB-

Services

for

attracting

-LRB-

e.g.

quiz

lottery

e-card

maps

weather

channels

download

service

-RRB-

C

Contents

for

attracting

-LRB-

e.g.

entertainments

culture

tourism

game

kids

health

gallery

-RRB-

-LRB-

-RRB-

Informing

consists

of

nine

items

developed

by

modifying

Simeon

's

-LRB-

-RRB-

components

local

links

contents

for

publicity

contents

for

learning

reports

descriptions

on

the

institution

descriptions

on

online

administrative

services

projects

contact

information

and

counseling

-LRB-

-RRB-

Community

consists

of

ten

items

online

forum

events

partner

links

-LRB-

or

ads

-RRB-

e-Magazine

-LRB-

or

newsletter

or

Webcast

-RRB-

message

boards

users

participation

-LRB-

e.g.

articles

photos

personal

links

-RRB-

focus

of

news

vision

-LRB-

or

values

-RRB-

domain

identity

and

community

services

-LRB-

or

online

support

for

community

meeting

or

networking

-RRB-

A

good

example

of

the

latter

is

the

Citizen

discussion

room

of

Ulsan

City

-LRB-

eulsan.go.kr

-RRB-

-LRB-

-RRB-

Delivering

as

a

variable

is

determined

by

the

presence

or

absence

of

features

like

search

engine

mailing

list

framework

multimedia

password

system

FAQ

chat

downloadable

publications

and

update

indication

-LRB-

-RRB-

Innovation

Public

institutions

have

to

utilize

the

Internet

for

actual

service

innovation

Hence

two

variables

indicating

innovation

results

are

selected

transformation

level

of

existing

services

and

frequency

of

new

innovative

services

These

are

each

rated

on

a

five-point

scale

-LRB-

-RRB-

never

-LRB-

-RRB-

only

descriptions

-LRB-

-RRB-

online

request

-LRB-

-RRB-

partial

-LRB-

-RRB-

full

processing

for

the

first

item

and

-LRB-

-RRB-

never

to

-LRB-

-RRB-

many

new

systems

for

the

second

item

Such

quantification

is

possible

because

the

introduction

of

new

innovative

systems

on

public

sector

Web

sites

is

growing

for

example

Docket

Access

View

Property

Assessments

and

Request

for

Proposals

of

Philadelphia

-LRB-

phila.gov

-RRB-

and

Citizen

Assessment

Systems

Citizen

Satisfaction

Monitor

Online

Procedures

Enhancement

-LRB-

OPEN

-RRB-

System

of

Seoul

-LRB-

metro.seoul.kr

-RRB-

These

measures

focus

mainly

on

components

visible

to

users

and

do

not

take

into

account

back-office

components

like

integration

Van

der

Merwe

and

Bekker

-LSB-

-RSB-

classify

website

evaluation

criteria

in

groups

as

shown

in

Many

of

their

criteria

seem

to

be

inspired

by

their

e-Commerce

orientation

but

many

of

the

criteria

will

be

applicable

to

e-Government

as

well

Table

Web

site

evaluation

criteria

groups

according

to

Merwe

and

Bekker

-LSB-

-RSB-

Phase

Criteria

group

This

criteria

group

evaluates/measures

Interface

Graphic

design

principles

The

effective

use

of

color

text

backgrounds

and

other

general

graphic

design

principles

Graphics

and

multimedia

The

effectiveness

of

the

graphics

and

multimedia

used

on

the

site

Style

and

text

Whether

or

not

the

text

is

concise

and

relevant

and

the

style

good

Flexibility

and

compatibility

The

degree

to

which

the

interface

is

designed

to

handle

exceptions

for

example

text-only

versions

of

pages

Navigation

Logical

structure

The

organization

and

menu

system

of

the

site

Ease

of

use

The

ease

of

navigation

to

find

the

pages

that

the

user

is

looking

for

Search

engine

The

search

engine

's

ability

to

find

the

correct

pages

easily

and

provide

clear

descriptions

of

the

search

results

Navigational

necessities

Other

important

aspects

of

navigation

like

the

absence

of

broken

links

and

under-construction

pages

Content

Product/service-related

information

Whether

or

not

the

products/services

are

described

precisely

and

thoroughly

Agency

and

contact

Information

Whether

or

not

it

is

easy

to

find

information

on

the

company

its

employees

and

its

principals

Information

quality

The

currency

and

relevance

of

the

content

on

the

site

Interactivity

How

much

input

the

user

has

on

the

content

displayed

on

the

site

Reliability

Stored

customer

profile

The

registering

process

and

how

the

company

uses

the

stored

customer

profile

Order

process

The

effectiveness

and

ease

of

use

of

the

online

order

process

After-order

to

order

receipt

The

company

's

actions

from

order

placement

until

the

order

is

delivered

Customer

service

How

the

company

communicates

and

helps

its

online

customers

Technical

Speed

Different

aspects

of

the

loading

speed

of

the

site

Security

Security

systems

and

the

ways

used

by

the

company

to

protect

customers

privacy

on

the

site

Software

and

database

Flexibility

in

terms

of

different

software

used

Also

looks

at

the

data

software

and

data

communication

systems

used

on

the

site

System

design

The

correct

functioning

of

the

site

and

how

well

it

integrates

with

internal

and

external

systems

Hazlett

and

Hill

discuss

-LSB-

-RSB-

discuss

the

current

level

of

government

measurement

Huang

and

Chao

-LRB-

-RRB-

note

that

while

the

development

and

management

of

web

sites

are

becoming

essential

elements

of

public

sector

management

little

is

known

about

their

effectiveness

Indeed

Kaylor

et

al.

-LRB-

-RRB-

note

that

research

into

the

effectiveness

of

e-Government

efforts

tends

to

concentrate

on

content

analysis

or

measures

of

usage

These

may

not

be

wholly

appropriate

metrics

with

which

to

gauge

success

Aspects

of

service

relevant

in

this

context

may

according

to

Voss

-LRB-

-RRB-

include

consumer

perceptions

of

security

and

levels

of

trust

response

times

-LRB-

bearing

in

mind

that

Internet

consumers

may

well

be

accustomed

to

quick

responses

-RRB-

navigability

of

the

Web

site

download

time

fulfillment

of

service

promised

timely

updating

of

information

site

effectiveness

and

functionality

Reinforcing

a

point

made

above

Voss

-LRB-

-RRB-

takes

the

view

that

e-service

channels

should

be

regarded

as

providing

support

for

front-line

service

delivery

and

not

replacements

for

front-line

service

However

such

channels

do

enable

change

in

the

nature

of

front-line

service

delivery

and

of

human

involvement

in

service

OVERHEID.NL

The

Dutch

Government

has

recently

published

Overheid.nl

Monitor

its

fifth

annual

e-government

progress

report

While

highlighting

a

number

of

encouraging

developments

the

report

concludes

that

much

remains

to

be

done

in

areas

such

as

user-friendliness

transactional

services

and

e-democracy

Overheid.nl

focused

on

all

government

agencies

and

mentions

the

following

agencies

explicitly

Municipalities

Ministries

Provinces

Water

boards

A

screenshot

of

the

online

monitor

of

the

measurement

of

municipalities

is

shown

in

the

figure

below

Figure

Screenshot

of

Overheid.nl

Overheid.nl

Monitor

developments

in

electronic

government

is

based

on

a

periodical

large-scale

survey

of

government

websites

which

was

carried

out

in

October

by

Advies

Overheid.nl

on

behalf

of

the

Dutch

Ministry

of

the

Interior

and

Kingdom

Relations

The

survey

assessed

government

websites

according

to

five

criteria

user-friendliness

general

information

government

information

government

services

and

scope

for

participation

-LRB-

interactive

policy

making

-RRB-

The

website

user

friendliness

measurement

etc

is

very

thorough

in

this

survey

The

e-service

measurement

is

less

well

defined

The

services

investigated

for

the

survey

are

listed

clearly

for

several

layers

of

government

but

they

seem

to

be

limited

to

the

so

called

Dutch

service

product

catalogue

set

of

typical

municipal

products

and

services

Figure

Sample

listing

of

services

measured

and

ways

of

accessing

them

investigated

Additionally

researchers

measured

the

e-mail

response

time

of

government

websites

and

assessed

user

satisfaction

via

a

survey

of

users

The

report

states

that

although

e-government

services

are

developing

on

schedule

and

are

becoming

more

sophisticated

there

is

still

much

room

for

improvement

On

the

positive

side

the

report

finds

that

E-government

is

developing

on

schedule

The

target

of

providing

of

government

services

electronically

was

widely

achieved

with

of

services

for

citizens

and

of

services

for

businesses

e-enabled

by

October

However

the

report

also

identifies

a

number

of

shortcomings

and

areas

where

improvement

is

needed

Practically

no

full

electronic

transactions

are

available

In

this

respect

the

report

considers

that

development

of

such

services

will

depend

both

on

future

solutions

for

user

identification

and

authentication

and

on

back-office

re-engineering

Although

the

use

of

e-services

is

growing

the

development

of

e-government

is

still

mainly

supply-driven

and

the

penetration

of

government

websites

remains

unknown

Only

if

we

assess

the

penetration

of

government

websites

and

the

level

of

their

use

can

we

take

a

truly

demand-driven

approach

the

report

says

The

items

related

to

municipalities

are

connected

to

the

functionalities

within

an

implemented

product

and

services

catalogue

Is

a

product

catalogue

or

another

form

of

systematically

offered

services

been

used

if

so

does

at

least

contain

products

if

so

does

is

at

least

contain

forms

i.e.

can

one

request

for

or

comment

on

at

least

product

by

using

a

form

that

is

contained

in

the

product

catalogue

which

can

be

filled

in

printed

and

send

in

by

the

users

if

so

can

these

be

accessed

per

life

event

if

so

can

these

be

accessed

per

theme

if

so

can

these

be

accessed

by

using

a

lexicon

-LRB-

a-z

-RRB-

if

so

does

it

contain

a

specific

search

function

-LRB-

fill

in

the

search

term

-RRB-

Besides

this

four

commonly

municipal

products

are

mentioned

that

can

be

supplied

more

or

less

in

a

digital

form

-LRB-

choices

no

info

info

down

loadable

form

up

loadable

form

transaction

-RRB-

Request

for

building

permission

Request

for

Cutting

trees

permission

Request

for

extract

from

GBA

Report

of

change

of

address

removal

-LRB-

no

transaction

possible

-RRB-

Users

satisfaction

with

e-government

services

is

still

significantly

lower

than

with

services

delivered

through

traditional

channels

E-democracy

tools

and

citizen

engagement

through

electronic

means

remain

embryonic

According

to

the

report

this

is

due

not

only

to

a

lack

of

demand

but

also

to

a

poorly

organized

supply

side

with

inadequate

moderation

unappealing

consultation

subjects

and

missing

follow-up

In

addition

to

identifying

progress

accomplished

and

remaining

issues

the

report

makes

a

number

of

recommendations

that

should

help

reach

the

objectives

of

the

Andere

Overheid

-LRB-

Other

Government

-RRB-

action

programme

presented

in

December

Such

recommendations

include

the

following

E-government

services

must

become

more

user-friendly

and

easier

to

find

Metadata

standards

should

be

defined

to

make

them

easier

to

find

through

search

engines

FAQs

and

lists

of

the

most

searched

terms

and

documents

should

also

be

made

more

widely

available

E-government

services

must

be

permanently

improved

even

once

all

government

websites

are

fully

functional

government

should

still

constantly

aim

to

improve

e-government

and

consult

target

groups

about

new

services

they

might

require

says

the

report

E-government

must

be

further

developed

through

service

integration

across

government

bodies

which

is

currently

still

in

its

infancy

According

to

the

report

the

Dutch

supply-driven

approach

has

so

far

sought

solutions

within

the

limits

and

administrative

competencies

of

single

bodies

Emphasis

must

be

shifted

from

the

breadth

of

services

to

their

depth

Rather

than

aiming

to

run

every

electronic

product

and

service

conceivable

government

bodies

should

aim

to

integrate

services

as

deeply

as

possible

especially

those

in

frequent

and

popular

demand

the

report

says

This

implies

developing

seamless

links

from

the

front

to

the

back

office

and

fostering

a

more

customer-minded

culture

From

the

perspective

of

the

policymakers

we

may

conclude

that

the

benchmark

takes

into

account

individual

agencies

and

their

websites

number

of

services

and

to

a

certain

degree

also

service

levels

but

the

aim

is

to

integrate

horizontally

something

which

is

not

measured

by

WEBDAM.NL

In

the

year

the

ministry

of

interior

decided

that

all

municipalities

should

be

online

by

and

of

the

services

provisioning

should

be

support

by

websites

In

order

to

help

municipalities

to

achieve

this

the

Webdam

project

was

started

in

March

aiming

at

stimulating

municipalities

to

develop

websites

These

websites

should

deliver

better

and

improved

services

provisioning

over

the

Internet

as

citizens

expected

One

of

the

activities

in

the

Webdam

project

has

been

the

development

of

a

website

that

allowed

municipalities

to

share

knowledge

and

make

better

use

of

the

Internet

To

further

stimulate

municipalities

Webdam

started

a

Top

for

municipalities

websites

using

criteria

such

as

design

content

service

level

and

communication

Assessment

and

ranking

of

each

municipality

is

performed

by

representatives

coming

from

three

groups

civil

servants

citizens

and

experts

Webdam

uses

a

panel

of

experts

to

judge

the

public

agencies

web

pages

The

stakeholders

include

the

following

groups

Webdam

employees

-LRB-

experts

-RRB-

Public

servants

municipality

under

study

Citizens

These

stakeholders

judges

the

web

pages

based

on

fie

main

groups

Layout

Content

Communication

services

and

plus/minus

remarks

Each

group

has

a

minimum

and

maximum

score

The

total

is

aggregated

to

determine

a

ranking

Who

determines

score

Figure

Screenshot

of

webdam

Webdam

focuses

exclusively

on

the

front-office

the

aspects

directly

visibility

to

the

citizens

using

the

web

pages

No

connection

to

the

size

of

the

municipality

the

number

of

citizens

and

other

expect

influencing

the

available

resources

of

the

municipality

is

made

Accenture

e-gov

monitor

The

yearly

research

conduct

by

Accenture

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

has

a

profound

influence

on

governments

An

increase

of

decrease

in

ranking

of

this

report

results

in

discussions

about

the

future

of

e-government

Accenture

researchers

in

each

of

the

selected

countries

described

the

typical

services

that

national

government

should

provide

using

the

Internet

in

order

to

fulfill

the

obvious

needs

of

citizens

and

businesses

They

accessed

and

assessed

the

websites

of

national

government

agencies

to

determine

the

quality

and

maturity

of

services

and

the

level

at

which

business

can

be

conducted

electronically

with

government

In

total

national

government

services

across

nine

major

service

sectors

were

investigated

by

Accenture

during

a

two

weeks

lasting

study

-LRB-

-RRB-

in

using

the

web

in

countries

The

nine

service

sectors

researched

were

Human

Services

Justice

Public

Safety

Revenue

Defence

Education

Transport

Motor

Vehicles

Regulation

Democracy

Procurement

and

Postal

The

main

indicator

of

the

eGovernment

level

chosen

by

Accenture

is

what

they

call

service

maturity

Service

maturity

is

an

indication

for

the

level

to

which

a

government

has

developed

an

online

presence

It

takes

into

account

the

numbers

of

services

for

which

national

governments

are

responsible

that

are

available

online

-LRB-

Service

Maturity

Breadth

-RRB-

and

the

level

of

completeness

with

which

each

service

is

offered

-LRB-

Service

Maturity

Depth

-RRB-

Service

Maturity

Overall

is

the

product

of

Service

Maturity

Breadth

and

Service

Maturity

Depth

Service

maturity

is

decomposed

into

the

following

aspects

Publish

Passive/Passive

Relationship

The

user

does

not

communicate

electronically

with

the

government

agency

and

the

agency

does

not

communicate

-LRB-

other

than

through

what

is

published

on

the

website

-RRB-

with

the

user

Interact

Active/Passive

Interaction

The

user

must

be

able

to

communicate

electronically

with

the

government

agency

but

the

agency

does

not

necessarily

communicate

with

the

user

Transact

Active/Active

Interaction

The

user

must

be

able

to

communicate

electronically

with

the

government

agency

and

the

agency

must

be

able

to

respond

electronically

to

the

user

the

degree

to

which

the

services

are

organized

around

the

citizen

as

opposed

to

around

internal

government

structures

In

Accenture

again

investigated

service

sectors

and

services

in

yet

again

two

weeks

They

were

agriculture

defence

e-Democracy

education

human

services

immigration

justice

and

security

postal

procurement

regulation

participation

revenue

and

customs

and

transport

Little

is

said

by

Accenture

about

the

metrics

involved

They

have

performed

the

survey

for

five

years

now

and

the

perspective

chosen

is

that

of

a

citizen

accessing

a

government

service

using

on

line

means

For

this

article

it

is

interesting

to

note

the

final

remarks

in

the

report

governments

are

at

the

end

of

what

can

be

achieved

with

traditional

methods

they

are

developing

strategies

to

cope

with

horizontal

integration

between

agencies

Regional

innovation

scorecard

-LRB-

ris

-RRB-

One

of

the

ambitions

of

the

EU

is

to

become

the

most

competitive

and

dynamic

knowledge-based

economy

of

the

world

-LRB-

Lissabon

agenda

-RRB-

To

measure

this

the

European

Regional

Innovation

Scorecard

-LRB-

RIS

-RRB-

a

scorecard

used

for

monitoring

and

comparing

the

innovation

in

regions

has

been

developed

-LSB-

-RSB-

The

scorecard

is

seen

as

acknowledged

instrument

to

compare

regions

in

their

ability

to

foster

economic

growth

The

Largest

province

of

The

Netherlands

The

Province

of

South

Holland

explicitly

states

on

their

website

The

province

of

Noord-Brabant

ranks

third

on

the

European

regional

innovation

scoreboard

Zuid-Holland

will

have

to

make

a

considerable

effort

in

the

coming

years

if

it

is

to

reach

the

They

also

state

that

the

scoreboard

is

regarded

as

extremely

relevant

because

it

is

generally

accepted

as

the

a

leading

European

benchmark

for

innovation

dynamics

-LSB-

-RSB-

Surprisingly

the

same

regional

authority

does

not

pay

any

attention

to

the

contribution

of

their

own

eGovernment

services

to

that

level

of

innovation

and

economic

growth

There

is

no

mentioning

of

eGovernment

or

government

services

or

anything

close

to

it

in

the

whole

policy

documents

related

to

innovation

at

this

Province

This

becomes

less

surprising

when

the

indicators

of

the

RIS

and

the

EIS

are

viewed

more

closely

The

RIS

uses

the

following

indicators

-LRB-

-RRB-

population

with

tertiary

education

-LRB-

-RRB-

lifelong

learning

-LRB-

-RRB-

employment

in

medium/high-tech

manufacturing

-LRB-

-RRB-

employment

in

high-tech

services

-LRB-

-RRB-

public

R&D

expenditures

-LRB-

-RRB-

business

R&D

expenditures

-LRB-

-RRB-

EPO

high-tech

patent

applications

-LRB-

-RRB-

all

EPO

patent

applications

and

-LRB-

-RRB-

five

indicators

using

unpublished

data

-LRB-

-RRB-

the

share

of

innovative

enterprises

in

manufacturing

and

services

innovation

expenditures

as

a

percentage

of

turnover

in

both

-LRB-

-RRB-

manufacturing

and

-LRB-

-RRB-

services

-LRB-

-RRB-

the

share

of

sales

of

new-to-the-firm

products

in

manufacturing

These

indicators

are

based

on

Eurostat

exercises

-LSB-

-RSB-

From

this

analysis

it

becomes

obvious

that

this

Province

considers

innovation

as

a

development

process

outside

the

government

and

its

own

performance

The

basic

assumption

made

by

the

Dutch

Provinces

is

that

governments

can

stimulate

innovation

in

the

economy

without

being

part

of

the

regional

economy

The

main

political

driver

for

efficient

eGovernment

is

economic

growth

and

jobs

and

the

main

driver

for

economic

growth

is

considered

to

be

innovation

The

metrics

of

the

benchmarks

do

no

coincide

The

evaluation

instruments

described

before

are

just

examples

of

the

overwhelming

number

measurement

instruments

currently

used

Table

summarizes

the

described

instruments

Although

we

focused

on

a

limited

number

of

instruments

these

instruments

are

very

likely

representative

for

the

other

measurement

instruments

The

following

observations

can

be

made

about

the

measurement

instruments

Most

instruments

focus

on

the

performance

of

a

single

agency

Measurement

focus

on

front

which

is

directly

visible

and

not

on

the

business

process

and

information

system

in

the

back

This

is

surprisingly

as

these

aspects

influence

the

performance

to

a

large

extend

Short-term

focus

not

many

indicators

are

focused

on

measuring

the

long-term

performance

Interpretation

of

measures

is

difficult

as

all

existing

measurement

instruments

lack

a

framework

depicting

the

relationships

between

the

indicators

and

the

use

of

resources

Different

stakeholders

may

come

to

different

interpretations

of

the

status

of

e-government

From

a

theoretical

point

of

view

we

conclude

after

examining

many

other

existing

instruments

that

these

instruments

lack

a

clear

connection

to

any

theoretical

framework

on

e-Government

and

a

well-described

set

of

preferences

that

can

be

used

for

comparison

Even

if

we

would

consider

that

these

measurements

instruments

were

developed

independent

of

each

other

it

is

astonishing

that

these

instruments

show

that

little

overlap

both

in

features

as

in

measurement

method

Table

Summary

of

measurement

instruments

studied

Governmental

performance

is

dependent

on

a

complex

of

interlinked

processes

and

dependencies

between

these

processes

the

stakeholders

involved

including

civil

servants

and

their

departments

The

legal

and

political

context

which

is

very

dominant

in

a

governmental

setting

furthermore

increases

complexity

Sometimes

obvious

improvements

in

a

service

provision

chain

may

be

blocked

because

data

may

not

be

shared

due

to

data

protection

regulations

The

system

of

checks

an

balances

that

is

fundamental

to

governments

functioning

and

essential

for

maintaining

citizens

trust

in

the

government

can

be

troublesome

if

we

want

to

redesign

inefficient

processes

A

combination

of

factors

such

as

the

volume

of

regulations

and

the

lack

of

understanding

of

their

original

aims

the

lack

of

formal

process

models

that

could

help

to

get

insights

in

the

dependencies

between

processes

and

explain

how

the

legal

requirements

are

translated

into

process

requirements

and

the

lack

of

formally

described

legal

models

do

n't

really

help

if

we

want

to

explicitly

formulate

the

criteria

that

determine

e-Government

success

These

criteria

determine

e-Government

success

-LRB-

or

failure

-RRB-

are

exactly

the

ones

that

should

be

the

ones

in

our

measurement

instruments

But

even

if

we

would

have

had

a

better

theory

on

performance

of

e-Government

processes

and

we

would

have

had

well

funded

measurement

instruments

interpretation

of

the

outcomes

of

applying

those

instruments

would

be

problematic

especially

within

the

political

context

within

these

instruments

are

generally

used

Bruin

-LSB-

-RSB-

showed

that

when

the

distance

between

the

interpreters

and

providers

of

information

is

bigger

it

is

more

difficult

to

interpret

information

Politicians

do

not

always

steer

on

rational

grounds

but

suppose

they

would

then

their

control

system

-LRB-

or

policy

making

process

see

figure

-RRB-

would

include

a

comparison

and

control

function

We

stated

before

that

comparison

is

based

upon

a

set

of

preferences

Public

services

can

consequently

be

evaluated

using

competing

norms

and

values

-LSB-

-RSB-

A

court

for

example

might

be

asked

to

deal

with

cases

as

efficiently

as

possible

to

maximize

the

number

of

cases

dealt

with

within

a

certain

time

period

on

the

one

hand

while

on

the

other

hand

the

sentence

should

be

carefully

made

funded

with

the

right

arguments

and

understandable

Performance

measurement

instruments

that

lack

an

explicit

set

of

preferences

-LRB-

or

norms

for

comparison

-RRB-

might

give

a

wrong

view

on

reality

if

looked

at

with

other

preferences

in

mind

We

investigate

the

current

e-government

measurement

practice

in

the

Netherlands

and

investigated

some

theoretical

work

in

this

field

Our

analyzes

shows

a

messy

picture

on

the

measurement

of

e-government

Many

measurement

instruments

take

a

too

simplistic

view

and

focus

on

measuring

what

is

easy

to

measure

Many

of

the

instruments

focus

on

measuring

the

visible

front

of

e-government

and

ignore

the

performance

of

the

cross-agency

business

processes

of

the

instruments

focus

on

measuring

multi-service

organizations

The

instruments

focus

on

one

-LRB-

type

of

-RRB-

agency

and

do

not

provide

an

overall

picture

Interpretation

of

measures

is

difficult

as

all

existing

measurement

instruments

lack

a

framework

depicting

the

relationships

between

the

indicators

and

the

use

of

resources

The

different

measures

may

fit

the

aim

of

the

owners

of

the

e-governmental

services

however

due

to

conflicting

aims

and

priorities

little

agreement

exists

on

a

uniform

set

of

measures

needed

for

comparison

of

e-government

development

Different

stakeholders

may

come

to

different

interpretations

of

the

status

of

e-government

As

such

the

existing

instruments

provide

a

picture

of

the

status

of

e-government

that

may

not

useful

as

surrogates

for

deducing

the

e-government

performance

Traditional

methods

of

measuring

e-government

impact

and

resource

usage

fall

short

of

the

richness

of

data

required

for

the

effective

evaluation

of

e-government

strategies

A

good

theoretical

framework

for

measuring

the

impact

of

e-government

and

the

use

of

resources

is

still

lacking

Due

to

this

fact

and

the

many

reports

that

are

produced

on

e-Government

developments

based

on

different

measurement

instruments

that

used

different

criteria

we

can

hardly

learn

from

existing

practices

It

would

be

Measurement

instrument

Focus

Update

frequency

Source

data

Characteristics

of

the

method

Overheid.nl

All

public

agency

websites

Yearly

Experts

Ranking

based

on

web

site

features

Webdam

Municipality

websites

Monthly

-LRB-

continuous

-RRB-

Expert

panel

consisting

of

types

of

representatives

-RRB-

Civil

servants

-RRB-

citizens

and

-RRB-

experts

Ranking

based

on

web

site

features

Accenture

Comparison

of

countries

Yearly

Accenture

researchers

based

on

judgment

of

a

selected

services

Ranking

based

inventory

of

services

Regional

innovation

scorecard

European

regions

Eurostat

Ranking

based

on

economic

quantitative

indicators

beneficial

for

both

citizens

as

for

governments

if

such

a

theoretical

framework

would

be

developed

and

a

more

or

less

standardized

measurement

instrument

could

become

available

This

would

allow

governments

and

designers

to

compare

different

e-government

approaches

and

learn

from

them

and

learning

from

our

experiences

certainly

fits

within

the

ambition

of

the

European

Union

to

become

the

most

competitive

and

dynamic

knowledge-based

economy

of

the

world

-LSB-

-RSB-

Accenture

-LRB-

-RRB-

Governments

Closing

Gap

Between

Political

Rhetoric

and

eGovernment

Reality

-LSB-

-RSB-

Accenture

-LRB-

-RRB-

eGovernment

Leadership

Realizing

the

Vision

-LSB-

-RSB-

Accenture

-LRB-

-RRB-

eGovernment

Leadership

Engaging

the

Customer

-LSB-

-RSB-

Armour

F.J.

Kaisler

S.H.

and

Liu

S.Y.

-LRB-

-RRB-

A

big-picture

look

at

Enterprise

Architectures

IEEE

IT

Professional

-LRB-

-RRB-

-LSB-

-RSB-

Bign

E.

Moliner

M.A.

and

S

nchez

J.

-LRB-

-RRB-

Perceived

Quality

and

satisfaction

in

multiservice

organizations

The

case

of

Spanish

public

services

Journal

of

Services

Marketing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

Boer

A.

Engers

T.

van

and

R.

Winkels

-LRB-

-RRB-

Using

Ontologies

for

Comparing

and

Harmonizing

Legislation

In

Proceedings

of

the

International

Conference

on

Artificial

Intelligence

and

Law

-LRB-

ICAIL

-RRB-

Edinburgh

-LRB-

UK

-RRB-

ACM

Press

-LSB-

-RSB-

Alexander

Boer

Radboud

Winkels

Rinke

Hoekstra

and

Tom

M.

van

Engers

Knowledge

Management

for

Legislative

Drafting

in

an

International

Setting

In

D.

Bourcier

editor

Legal

Knowledge

and

Information

Systems

Jurix

The

Sixteenth

Annual

Conference.

pages

Amsterdam

IOS

Press

-LSB-

-RSB-

Bons

R.

Ronald

M.Lee

and

Tan

Yua-Hua

-LRB-

-RRB-

A

Formal

Specification

of

Automated

Auditing

of

Trustworthy

Trade

Procedures

for

Open

Electronic

Commerce

Hawaii

International

Conference

on

System

Sciences

-LRB-

HICCS

-RRB-

-LSB-

-RSB-

Buckland

and

F.

Gey

-LRB-

-RRB-

The

relationship

between

recall

and

precision

Journal

of

the

American

Society

for

Information

Science

-LRB-

-RRB-

-LSB-

-RSB-

Bruin

H.

de

-LRB-

-RRB-

Performance

measurement

in

the

public

sector

strategies

to

cope

with

the

risks

of

performance

measurement

The

International

Journal

of

Public

Sector

Management

vol

no.

pp.

-LSB-

-RSB-

Checkland

P.

-LRB-

-RRB-

Systems

Thinking

Systems

Practice

Wiley

Chichester

-LSB-

-RSB-

Coase

R.

-LRB-

-RRB-

The

Nature

of

the

Firm

Economia

-LSB-

-RSB-

European

Commissions

-LRB-

-RRB-

European

Innovation

Scoreboard

European

Trend

Chart

on

Innovation

Innovation/SMEs

Programme

-LSB-

-RSB-

European

Commission

-LRB-

-RRB-

Green

paper

on

Public

private

partnerships

and

community

law

on

public

contracts

and

concessions

European

Commission

no.

-LSB-

-RSB-

Fagan

J.C.

Fagan

B.

-LRB-

-RRB-

An

accessibility

study

of

state

legislative

web

sites

Government

Information

Quarterly

-LSB-

-RSB-

Galliers

R.D.

-LRB-

-RRB-

Information

Systems

Research

Issues

methods

and

practical

guidelines

Alfred

Waller

Fawley

England

-LSB-

-RSB-

Green

S.H.B.

-LRB-

-RRB-

Cyberspace

winners

how

they

did

it

Business

Week

June

pp.

-LSB-

-RSB-

Groot

H.

de

and

R.

Goudriaan

-LRB-

-RRB-

De

productiviteit

van

de

overheid

over

prestaties

personeel

en

uitgaven

in

de

publieke

sector

Academic

Service

Schoonhoven

The

Netherlands

-LSB-

-RSB-

Hazlett

S.A.

and

Hill

F.

-LRB-

-RRB-

E-government

the

realities

of

using

IT

to

transform

the

public

sector

Managing

Service

Quality

Vol

No.

pp.

-LSB-

-RSB-

Janssen

M.F.W.H.A.

-LRB-

-RRB-

Designing

Electronic

Intermediaries

Doctoral

Dissertation

Delft

University

of

Technology

-LSB-

-RSB-

Janssen

Marijn

Davidse

Anouk

-LRB-

-RRB-

Evaluation

of

a

Performance-based

Accountability

System

The

European

Conference

on

E-government

-LRB-

ECEG

-RRB-

Dublin

Castle

Dublin

Ireland

June

-LSB-

-RSB-

Jensen

M.

and

Meckling

W.

-LRB-

-RRB-

Theory

of

the

Firm

Managerial

behavior

agency

costs

and

capital

structure

Journal

of

Financial

Economics

-LSB-

-RSB-

Layne

KJL

Lee

J.

-LRB-

-RRB-

Developing

fully

functional

E-government

A

four

stage

model

Government

Information

Quarterly

Vol

No.

pp

-LSB-

-RSB-

Lee

J.K.

-LRB-

-RRB-

A

model

for

monitoring

public

sector

web

site

strategy

Internet

Research

Electronic

networking

application

and

policy

Vol

no.

-LSB-

-RSB-

Malone

T.W.

Crowston

K.

-LRB-

-RRB-

The

Interdisciplinary

Study

of

Coordination

ACM

Computing

Surveys

vol

no.

pp.

-LSB-

-RSB-

Merwe

R.

van

der

and

Bekker

J.

-LRB-

-RRB-

A

framework

and

methodology

for

evaluating

e-commerce

web

sites

Internet

Research

electronic

Networking

Applications

and

Policy

Vol

No.

pp.

-LSB-

-RSB-

Moon

M.J.

-LRB-

-RRB-

The

Evolution

of

E-Government

Among

Municipalities

Rhetoric

or

reality

Public

Administration

Review

Vol

no.

pp.

-LSB-

-RSB-

Parasuraman

A.

Zeithaml

V.A.

and

Berry

L.L.

-LRB-

-RRB-

SERVQUAL

a

multiple

item

scale

for

measuring

consumer

perceptions

of

service

quality

Journal

of

Retailing

Vol

pp.

-LSB-

-RSB-

Peters

Rob

and

Wilson

Frank

-LRB-

-RRB-

Natural

Language

access

to

regional

information

sources

the

PortofRotterdam

case

International

Workshop

on

Image

Analysis

for

Multimedia

Interactive

Services

WIAMIS

-LSB-

-RSB-

Provincial

council

-LRB-

-RRB-

Innovatiebrief

kenniseconomie

Zuid-Holland

Kennismaken

met

Kenniszaken

page

-LSB-

-RSB-

Rohleder

S.J.

et

al.

-LRB-

-RRB-

eGovernment

Leadership

High

Performance

Maximum

Value

Fifth

Annual

Accenture

eGovernment

Study

Accenture

Government

Executive

Studies

-LSB-

-RSB-

Simeon

R.

-LRB-

-RRB-

Evaluating

domestic

and

international

Web

site

strategies

Internet

Research

Vol

No.

pp.

-LSB-

-RSB-

Quinn

R.E.

and

Rohrbaugh

J.W.

-LRB-

-RRB-

A

Spatial

Model

of

Effectiveness

criteria

Towards

a

competing

values

approach

to

organizational

effectiveness

Management

Science

-LSB-

-RSB-

Van

Engers

T.M.

Legal

Engineering

A

Knowledge

Engineering

Approach

To

Improving

Legal

Quality

in

eGovernment

and

eDemocracy

Progress

and

Challenges

Padget

J.

Neira

R.

De

Le

n

J.L.

Editors

Instituto

Polit

chnico

Nacional

Centro

de

Investigacion

en

Computaci

n

ISBN

-LSB-

-RSB-

Williamson

O.E.

-LRB-

-RRB-

Market

and

Hierarchies

Analysis

and

Antitrust

Implications

A

study

in

the

economics

of

internal

organization

Macmillan

New

York

-LSB-

-RSB-

Yin

R.K.

-LRB-

-RRB-

Case

Study

Research

Design

and

methods

Sage

publications

Newbury

Park

California

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

architectures
evaluation
benchmark
business process
law
measurement
public administration


Topic

Modeling

in

Fringe

Word

Prediction

for

AAC

Word

prediction

can

be

used

for

enhancing

the

communication

ability

of

persons

with

speech

and

language

impair-ments

In

this

work

we

explore

two

methods

of

adapting

a

language

model

to

the

topic

of

conversation

and

apply

these

methods

to

the

prediction

of

fringe

words

I.

m

Document

and

Text

Processing

Miscellaneous

Algorithms

Alternative

and

Augmentative

Communication

-LRB-

AAC

-RRB-

is

the

field

of

research

concerned

with

finding

ways

to

help

those

with

speech

difficulties

communicate

more

easily

and

completely

Today

there

are

approximately

million

people

in

the

United

States

with

some

form

of

communication

difficulty

One

means

to

help

ease

communication

is

the

use

of

an

electronic

communication

device

which

may

have

synthetic

speech

as

output

However

one

issue

in

using

an

AAC

device

is

communication

rate

Whereas

speaking

rate

is

estimated

at

words

per

minute

-LRB-

wpm

-RRB-

many

AAC

users

communication

rates

are

lower

than

wpm

-LSB-

-RSB-

Thus

one

goal

of

developers

is

to

find

ways

to

increase

the

rate

of

communication

by

making

AAC

devices

easier

to

use

and

more

intelligent

Some

researchers

have

attempted

to

speed

communication

rate

by

providing

quick

access

to

the

core

vocabulary

the

relatively

small

set

of

frequently

used

words

Methods

for

doing

this

include

abbreviation

expansion

and

iconic

methods

such

as

semantic

compaction

-LSB-

-RSB-

In

contrast

in

this

work

we

attempt

to

speed

access

to

the

much

larger

set

of

words

often

called

fringe

vocabulary

This

set

is

of

interest

because

although

each

individual

word

occurs

less

frequently

the

set

of

fringe

words

on

the

whole

is

very

significant

Suppose

that

the

user

wants

to

enter

I

want

a

home

in

the

country

After

typing

I

want

a

h

they

might

see

something

like

shown

below

The

system

has

created

a

prediction

window

containing

the

five

words

that

it

thinks

the

user

may

be

trying

to

type

In

this

example

the

user

can

press

to

complete

the

word

home

and

the

system

will

Copyright

is

held

by

the

author/owner

IUI

January

February

Sydney

Australia

ACM

enter

the

word

with

a

space

afterwards

So

in

this

example

the

user

needed

keystrokes

to

enter

what

would

normally

take

keystrokes

I

want

a

h

hundred

-LRB-

-RRB-

half

-LRB-

-RRB-

house

-LRB-

-RRB-

hard

-LRB-

-RRB-

home

-LRB-

-RRB-

It

is

difficult

to

judge

how

much

word

prediction

can

speed

communication

rate

Much

of

this

determination

is

dependent

on

the

accuracy

of

the

prediction

method

the

characteristics

of

the

user

such

as

their

physical

and

cognitive

abilities

and

the

characteristics

of

the

user

interface

such

as

where

the

prediction

list

is

displayed

and

how

a

word

in

the

list

is

selected

Here

the

prediction

method

is

evaluated

separately

from

the

rest

of

a

word

prediction

system

by

simulating

what

a

user

would

type

in

a

conversation

if

he/she

were

taking

full

advantage

of

the

prediction

list

This

theoretical

evaluation

measures

the

percentage

of

keystrokes

that

were

saved

by

word

prediction

over

typing

out

every

character

In

this

paper

we

first

describe

related

work

and

give

some

background

in

statistical

approaches

to

word

prediction

We

present

approaches

to

topic

modeling

and

compare

the

results

of

topic

modeling

to

a

baseline

method

For

a

more

thorough

account

of

this

work

visit

Several

previous

researchers

have

used

n-gram

models

in

word

prediction

for

AAC

-LSB-

-RSB-

For

example

Lesher

et

al.

-LSB-

-RSB-

show

how

increasing

training

set

size

and

unigrams

to

bigrams

-LRB-

going

from

to

-RRB-

to

trigrams

-LRB-

another

-RRB-

These

evaluations

used

a

window

size

of

Other

researchers

have

integrated

grammatical

information

into

n-gram

word

prediction

systems

Garay-Vitoria

and

Gonzalez-Abascal

-LSB-

-RSB-

integrated

a

statistical

chart

parser

while

Fazly

and

Hirst

-LSB-

-RSB-

and

Copestake

-LSB-

-RSB-

used

part-of-speech

-LRB-

POS

-RRB-

tagging

These

yielded

improvements

of

keystroke

savings

There

have

been

several

attempts

at

topic

modeling

in

the

language

modeling

community

particularly

for

speech

recognition

-LSB-

-RSB-

Some

of

the

evaluations

of

topic

modeling

have

found

different

variants

of

it

to

be

very

beneficial

-LSB-

-RSB-

Lesher

and

Rinkus

-LSB-

-RSB-

is

an

attempt

at

topic

modeling

for

word

prediction

but

does

not

use

dynamic

topic

modeling

like

-LSB-

-RSB-

and

this

work

Window

Bigrams

Trigrams

Method

A

Method

B

Table

The

keystroke

savings

of

topic

modeling

is

shown

compared

to

a

bigram

and

trigram

baseline

Like

several

of

the

aforementioned

word

prediction

researchers

we

use

n-gram

methods

for

language

modeling

Our

baseline

word

prediction

methods

use

bigram

and

trigram-based

n-gram

models

with

backoff

with

Good-Turing

smoothing

the

current

best

practice

in

statistical

language

modeling

according

to

Manning

and

Sch

utze

-LSB-

-RSB-

Additionally

we

incorporate

a

special

unigram

model

for

the

first

word

of

each

sentence

In

word

prediction

these

language

models

our

used

to

rank

all

the

words

that

the

user

could

possibly

be

typing

The

top

W

words

are

presented

to

the

user

where

W

is

the

prediction

window

size

Statistical

approaches

require

a

collection

of

text

to

construct

a

language

model

Ideally

our

corpus

would

be

a

large

collection

of

conversations

involving

one

or

more

people

using

an

AAC

system

Such

a

corpus

is

unavailable

so

we

follow

-LSB-

-RSB-

in

using

the

Switchboard

corpus

which

is

a

collection

of

telephone

conversations

and

their

transcriptions

The

training

section

contains

a

randomly

pre-selected

conversations

and

the

testing

section

contains

the

remaining

conversations

We

perform

preprocessing

to

remove

some

speech

repairs

in

accordance

with

Hindle

-LSB-

-RSB-

These

editing

rules

bring

the

Switchboard

conversations

closer

to

what

we

envision

an

AAC

user

would

type

Evaluation

We

compare

the

number

of

keystrokes

required

for

a

user

taking

full

advantage

of

our

word

prediction

system

to

the

number

of

keystrokes

required

to

enter

each

character

of

the

conversation

We

use

immediate

prediction

for

our

evaluations

which

allows

use

of

the

prediction

list

before

the

first

character

of

a

word

has

been

entered

We

assume

that

one

keystroke

is

required

to

speak

each

turn

of

input

and

that

a

space

is

automatically

inserted

after

a

word

is

selected

from

the

prediction

list

KS

keys

normal

keys

withprediction

keys

normal

Because

we

are

interested

in

the

prediction

of

fringe

words

our

evaluations

are

measured

on

fringe

words

only

Core

words

are

excluded

from

the

list

of

predictions

The

particular

core

vocabulary

we

chose

is

available

from

the

AAC

Centers

at

the

University

of

Nebraska

at

Lincoln

available

from

We

used

the

Young

Adult

Conversation

Regular

core

vocabulary

list

as

it

is

the

most

similar

to

the

type

of

conversations

in

the

Switchboard

corpus

The

Switchboard

transcriptions

were

available

from

The

goal

of

topic

modeling

is

to

identify

the

current

topic

of

conversation

then

increase

the

probability

of

related

words

and

decrease

the

probability

of

unrelated

words

Some

words

will

be

unaffected

by

topic

modeling

such

as

function

words

which

are

used

similarly

in

all

topics

It

is

for

this

reason

that

we

chose

to

improve

fringe

word

prediction

with

topic

modeling

we

feel

that

topic

modeling

specifically

improves

fringe

word

prediction

Researchers

are

consistent

in

representing

a

topic

by

creating

a

collection

of

representative

text

of

the

topic

However

researchers

differ

on

the

best

way

to

organize

a

collection

of

topics

Some

researchers

have

created

a

hierarchical

collection

of

topics

-LSB-

-RSB-

while

others

have

created

a

disjoint

set

of

topics

-LSB-

-RSB-

We

feel

that

the

primary

lure

of

a

hierarchical

approach

the

ability

to

generalize

can

be

captured

in

the

set

approach

as

well

by

giving

varying

weight

to

all

topics

and

not

just

the

most

likely

topic

For

this

reason

we

represent

topics

as

disjoint

sets

of

conversations

The

current

topic

of

conversation

must

be

identified

from

the

part

of

the

conversation

that

has

taken

place

so

far

and

updated

periodically

in

the

conversation

Thus

we

must

devise

a

representation

for

a

partial

conversation

for

assessing

the

similarity

of

the

conversation

to

each

topic

In

representing

the

conversation

so

far

we

choose

to

implement

an

exponentially

decayed

cache

like

-LSB-

-RSB-

using

TF-IDF

values

rather

than

raw

frequencies

This

follows

the

work

of

Mahajan

et

al.

-LSB-

-RSB-

in

considering

the

inverse

document

frequency

of

a

word

to

proportional

to

its

utility

in

identifying

the

current

topic

Because

our

approach

is

for

topic

identification

we

ignore

words

that

occur

in

or

more

of

the

topics

with

the

intuition

that

such

words

are

irrelevant

to

selection

of

topic

As

a

step

to

convert

our

model

of

the

current

conversation

to

a

model

of

the

current

topic

we

compute

the

document

similarity

between

the

cache

and

the

unigram

model

for

each

topic

We

chose

to

use

the

cosine

metric

following

-LSB-

-RSB-

Given

that

we

have

computed

a

similarity

score

between

each

topic

and

the

current

conversation

there

are

two

main

variations

on

how

to

construct

a

new

language

model

Mahajan

et

al.

-LSB-

-RSB-

implemented

a

k-nearest

solution

constructing

the

topic

model

from

the

most

similar

k

topics

Each

topic

's

language

model

was

weighted

equally

for

their

experiments

Instead

we

chose

to

follow

Florian

and

Yarowsky

's

approach

-LSB-

-RSB-

They

expand

the

probability

for

a

word

-LRB-

w

-RRB-

given

a

history

-LRB-

h

-RRB-

as

follows

P

-LRB-

w

h

-RRB-

X

ttopics

P

-LRB-

t

h

-RRB-

P

-LRB-

w

t

h

-RRB-

P

-LRB-

w

t

h

-RRB-

is

simply

the

probability

of

w

taken

from

the

language

model

constructed

for

topic

t

The

probability

of

the

topic

is

estimated

as

follows

P

-LRB-

t

h

-RRB-

S

-LRB-

t

h

-RRB-

P

t

topics

S

-LRB-

t

h

-RRB-

where

S

-LRB-

t

h

-RRB-

is

the

cosine

similarity

of

the

topic

to

the

current

part

of

the

conversation

Method

A

Our

first

method

of

topic

modeling

is

most

similar

in

spirit

to

the

work

of

Mahajan

et

al.

-LSB-

-RSB-

and

Florian

and

Yarowsky

-LSB-

-RSB-

In

training

a

bigram

model

is

computed

for

each

topic

in

Switchboard

In

testing

the

cache

representation

of

the

current

conversation

is

compared

against

the

unigram

representation

of

each

topic

and

similarity

scores

are

computed

The

similarity

scores

are

then

used

to

weight

the

frequencies

obtained

from

each

topic

in

a

linear

interpolation

Then

this

interpolated

bigram

model

is

used

to

compute

the

probabilities

used

for

word

prediction

Topic

modeling

shows

a

sizable

improvement

over

the

the

bigram

baseline

We

've

included

the

comparison

to

a

bigram

baseline

because

it

is

the

most

natural

baseline

in

terms

of

language

understanding

However

a

trigram

baseline

is

also

a

natural

comparison

when

considering

that

it

can

run

with

the

same

or

less

computational

resources

than

topic

modeling

When

compared

against

the

trigram

baseline

the

topic

model

gives

improvement

Method

B

Our

second

method

of

topic

modeling

is

more

similar

to

the

work

of

Bellegarda

-LSB-

-RSB-

Like

Bellegarda

we

compute

topic-dependent

unigram

probabilities

These

topic-dependent

probabilities

are

multiplied

with

probabilities

from

a

trigram

backoff

model

Additionally

we

weight

the

topic

component

with

a

tuning

parameter

After

manual

tuning

on

a

two

conversations

we

found

that

worked

well

Method

B

is

an

improvement

over

a

trigram

baseline

but

only

a

minor

improvement

We

feel

that

the

problem

is

that

a

low

value

was

necessary

to

avoid

overriding

the

word

preference

that

is

due

to

context

but

that

it

also

reduced

the

ability

of

the

overall

model

to

adapt

to

a

particular

topic

Comparison

Method

A

offers

an

additional

or

more

keystroke

savings

over

Method

B

for

most

window

sizes

This

is

due

to

the

low

weight

of

the

tuning

parameter

for

Method

B

However

as

previously

mentioned

the

low

weight

was

necessary

Additionally

notice

that

Method

A

becomes

comparatively

better

as

the

window

size

is

increased

The

trigram

model

component

in

Method

B

can

be

thought

of

as

a

stronger

source

of

knowledge

than

the

interpolated

bigram

model

of

Method

A.

Because

of

this

when

the

trigram

history

exists

in

the

language

model

Method

B

's

predictions

are

more

accurate

However

because

the

trigram

model

is

sparse

it

can

only

contribute

to

the

top

few

predictions

Thus

it

has

a

much

greater

effect

on

the

top

few

window

sizes

For

real

world

systems

however

absolute

performance

is

not

the

only

factor

The

computational

demands

of

each

approach

are

often

considered

when

selecting

a

practical

solution

The

trigram

baseline

processed

at

words

per

minute

-LRB-

wpm

-RRB-

Method

A

processed

conversations

in

testing

at

wpm

and

Method

B

processed

words

per

minute

Method

B

uses

barely

more

processing

time

than

the

trigram

baseline

model

Topic

modeling

can

be

implemented

in

many

different

ways

We

've

demonstrated

two

such

methods

for

topic

modeling

one

for

computationally

limited

devices

and

another

for

computationally

rich

devices

Both

methods

show

a

clear

improvement

over

a

trigram

model

with

backoff

Before

the

advent

of

word

prediction

a

user

would

've

pressed

keys

per

fringe

word

on

average

Now

with

topic

modeling

for

word

prediction

only

keys

per

word

are

required

We

would

like

to

thank

the

US

Department

of

Education

for

funding

this

research

under

grant

under

the

National

Institute

on

Disability

and

Rehabilitation

Research

program

We

would

also

like

to

thank

Dr.

Gregory

Lesher

for

correspondence

regarding

his

work

and

Dr.

David

Saunders

for

lending

us

a

compute

server

-LSB-

-RSB-

B.

Baker

Minspeak

Byte

pages

-LSB-

-RSB-

J.

Bellegarda

Large

vocabulary

speech

recognition

with

multispan

language

models

IEEE

Trans

On

Speech

and

Audio

Processing

-LRB-

-RRB-

-LSB-

-RSB-

D.

R.

Beukelman

and

P.

Mirenda

Augmentative

and

alternative

communication

Management

of

severe

communication

disorders

in

children

and

adults

P.H.

Brookes

Pub

Co.

-LSB-

-RSB-

L.

Boggess

Two

simple

prediction

algorithms

to

facilitate

text

production

In

Proceedings

of

the

second

conference

on

Applied

natural

language

processing

pages

Morristown

NJ

USA

Association

for

Computational

Linguistics

-LSB-

-RSB-

A.

Carlberger

J.

Carlberger

T.

Magnuson

M.

S.

Hunnicutt

S.

Palazuelos-Cagigas

and

S.

A.

Navarro

Profet

a

new

generation

of

word

prediction

An

evaluation

study

In

Proceedings

of

Natural

Language

Processing

for

Communication

Aids

-LSB-

-RSB-

S.

Chen

K.

Seymore

and

R.

Rosenfeld

Topic

adaptation

for

language

modeling

using

unnormalized

exponential

models

In

Proc

Int

l

Conf

on

Acoustics

Speech

and

Signal

Processing

-LSB-

-RSB-

A.

Copestake

Augmented

and

alternative

nlp

techniques

for

augmentative

and

alternative

communication

In

Proceedings

of

the

ACL

workshop

on

Natural

Language

Processing

for

Communication

Aids

-LSB-

-RSB-

A.

Fazly

and

G.

Hirst

Testing

the

efficacy

of

part-of-speech

information

in

word

completion

In

Proceedings

of

the

Conference

of

the

European

Chapter

of

the

Association

for

Computational

Linguistics

-LSB-

-RSB-

R.

Florian

and

D.

Yarowsky

Dynamic

nonlocal

language

modeling

via

hierarchical

topic-based

adaptation

In

Proceedings

of

ACL

pages

-LSB-

-RSB-

N.

Garay-Vitoria

and

J.

Gonz

alez-Abascal

Intelligent

word-prediction

to

enhance

text

input

rate

In

Proceedings

of

the

second

international

conference

on

Intelligent

User

Interfaces

-LSB-

-RSB-

D.

Hindle

Deterministic

parsing

of

syntactic

non-fluencies

In

Proceedings

of

the

Annual

Meeting

of

the

Association

for

Computational

Linguistics

-LSB-

-RSB-

G.

Lesher

B.

Moulton

and

J.

Higgonbotham

Effects

of

ngram

order

and

training

text

size

on

word

prediction

In

Proceedings

of

the

RESNA

Annual

Conference

-LSB-

-RSB-

G.

Lesher

and

G.

Rinkus

Domain-specific

word

prediction

for

augmentative

communication

In

Proceedings

of

the

RESNA

Annual

Conference

-LSB-

-RSB-

M.

Mahajan

D.

Beeferman

and

X.

D.

Huang

Improved

topic-dependent

language

modeling

using

information

retrieval

techniques

In

Proceedings

of

the

International

Conference

on

Acoustics

Speech

and

Signal

Processing

-LSB-

-RSB-

C.

Manning

and

H.

Sch

utze

Foundations

of

Statistical

Natural

Language

Processing

MIT

Press

-LSB-

-RSB-

A.

Newell

S.

Langer

and

M.

Hickey

The

r

ole

of

natural

language

processing

in

alternative

and

augmentative

communication

Natural

Language

Engineering

-LRB-

-RRB-

-LSB-

-RSB-

K.

Seymore

and

R.

Rosenfeld

Using

story

topics

for

language

model

adaptation

In

Proceedings

of

Eurospeech

pages

Rhodes

Greece

-LSB-

-RSB-

A.

L.

Swiffin

J.

A.

Pickering

J.

L.

Arnott

and

A.

F.

Newell

Pal

An

effort

efficient

portable

communication

aid

and

keyboard

emulator

In

Proceedings

of

the

Annual

Conference

on

Rehabilitation

Techonology

pages

B I O O B I O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I I I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

core vocabulary
prediction of fringe words
fringe vocabulary
communication rate
immediate prediction
language modeling
word prediction
prediction window size
conversations in the switchboard corpus
aac
construct a new language model
topic modeling

decrease probability of unrelated words
accuracy of prediction method
identify current topic of conversation
increase probability of related words