DESIGN OF INTERACTIVE SYSTEMS

DESIGN OF INTERACTIVE SYSTEMS Overheads Set I by Murray Turoff Department of Computer and Information Science New Jersey Institute of Technology Newark NJ, 07102 TEL: 973 596 3399 email: turoff@vc.njit.edu homepage://eies.njit.edu/~turoff/ © Copyright 1998 Murray Turoff CONTENTS

Subject One: Fundamental Concepts, Observations and Problems
Subject Two: Dimensions of Interface Design
Subject Three: The Design Process
Subject Four: Protocol Analysis
Subject Five: Indexing & Searching

FUNDAMENTAL CONCEPTS Subject One

Where is the wisdom we lost in knowledge?

Where is the knowledge we lost in information?

T. S. Eliot (The Rock)

ATMOSPHERE

HUMAN COMPUTER

Slow Rapid
Sloppy Rigorous
Forgetful Precise
Implicit Explicit
Subjective Objective
Brilliant Stupid

How do we design a communication process between the two? WHY INTERACTIVE I

Iterative Problem Solving
Unpredictable Sequences
Tool Flexibility
Impossible Manually

Creativity and Quality

Collaboration, Coordination & Communications

WHY INTERACTIVE II

Enhancement (save time and/or effort)
Speed, quantity, memory
Enjoyment & Challenge
Sublimating & Escape
Power & Control
Solving the Wrong Problem

INFORMATION DOMAINS OF USERS I

Single Function Tasks

simple inquiry / calculations / messaging

Structuring Tasks

Organizing / filtering / summarizing

Status Briefing / Report Generation
Tracking / Monitoring Tasks
Creation / Composition / Modeling

INFORMATION DOMAINS OF USERS II

Challenges for Individual Support

Exception Reporting
Diagnosis
Discovery
Hypothesis Testing and Analysis

Challenges for Group Support

Planning, Decision Analysis and Resource Allocation
Command, Control & Coordination

STAGES OF USER EVOLUTION

Uncertainty Phase
Insight Phase
Incorporation Phase
Saturation Phase

USER CHARACTERISTICS

High Cognitive Variability, Problem Solvers, Creative, Demanding
Novice, Casual, Intermediaries, Operators, Routine
Experienced, Frequent, Power
Children to Adults
Clerk, Analyst, Engineer, Scientist, Manager, Executive
Results:

Different User Roles in one System
Multiple Interface Methods Required

DESIGN METHODS I

Comparison/Differentiating
Designing / Requirements Analysis

Task / Macro
Cognitive / Micro
Group / Micro & Macro
Process Design / Integration
Enterprise Design/Organizational
Technology / Environmental

DESIGN METHODS II

Enhancements / Evolution
Visioning / Normative Planning

Social Engineering
Goal Analysis
Enterprise Design

THE DEVELOPMENT PROCESS

MODELS I

Mental Models of users resulting from their "real" world experiences
Requirements Model determined by the system analysis process acting on user mental models.
Implementation Model is the actual internal design model of the supporting software processes.

MODELS II

Real World is the actual system being model.
Interface Model/Metaphors is the mediator between the users mental model and the systems implementation model.

PROCESSES I

Comparison of the Implementation model to the

Requirements model is TESTING
Mental Model is EVALUATION
Real World is VALIDATION
Interface Model is TRAINING/LEARNING

PROCESSES II

Functional Opacity is created by large differences between the Interface Model and the Mental Model
System Opacity is created by large differences between the Interface Model and the Implementation Model.
Extreme functional opacity or system opacity can lead to cognitive dissonance.

VIRTUALITY Virtuality is the process whereby the implementation model in the computer replaces the real world model in a system made up in part of people.

Information Systems are usually design on a descriptive basis; however, they often become prescriptive.
Designers need to anticipate prescriptive impacts of their systems.

The design of an interactive system can be the design of a social system

GENERAL SYSTEMS VIEW

Interactive Systems are open, dynamic, and adaptive in the long run
They will be changed by the organizational/social environment and they will influence changes in those environments.

Law of Requisite Variety Applies: The inputs to a system must be as rich/variable as the outputs if one wishes to control the outputs of a system. FUNDAMENTAL PROBLEM I

Proper Level of Tools / Objects / Data Structure

Too Micro / Low Level

Difficult to work with
More effort and time

Too Macro / High Level

Inflexible
Difficult to Change

FUNDAMENTAL PROBLEM II

Gulf of Execution

Translating goals, intentions to actions
Functional Opacity

Gulf of Evaluation

Translating output display to evaluation
Not System Opacity but related
Excessive trial and error

PLAYING MUSIC PROBLEM

What are the principle alternative ways to make music?
Designing a system any idiot can use and only idiots will use it!
Novice Friendly, Experienced Hostile
Two Separate Dimensions

Beginning users / Experienced users
Non Domain Experts / Domain Experts

COMMON PSYCHOLOGICAL EFFECTS

Fishbowl
Bully
Peephole
Concrete
Clutter (Confusion)
People Angst
Computer Angst
Rorschach Blot

PSYCHOLOGICAL ROLES (Rorschach Blots) OF COMPUTERS Evaluator, Magician, Helper, Entertainer, Companion, Challenger, Foe, Mentor, Accomplice, Producer, Overseer, Dictator, Priest, Servant, Picky Parent, Unruly Child, God, etc.

Computers can be made to appear to be emotional and/or intelligent.
Providing computer literacy almost an ethical issue

USER RESPONSE TO INADEQUATE SYSTEM

DIS-USE: Turn to other sources
MIS-USE: Using inappropriate ways
PARTIAL USE: Use of wrong subset
DISTANT USE: Use of intermediary
MINIMUM USE: Recipe (IRS type) interaction
NON-USE: Avoiding the system
MODIFICATION OF TASK: Change the task to fit the system (solving wrong problem)
COMPENSATORY ACTIVITY: User has to do more
DIRECT PROGRAMMING: User modifies system/System encourages unique interfaces

Sources of Knowledge and Wisdom I

General Social Sciences Methods

Controlled Experiments
Focus Groups
Interview Methods
Participant Observation
Survey Methods
Longitudinal studies
Field Studies

Sources of Knowledge and Wisdom II

Industrial Engineering

Human Factor studies
Physiological response studies
"time and motion" studies (Method)
Human Monitoring (Method)

Psychology

Human cognitive processes
Human problem solving
Protocol Analysis (Method)
Human psychology and mental ability measurement

Sources of Knowledge and Wisdom III

Management Sciences

Cost/Benefit Analysis (Method)

Efficiency, Effectiveness, Quality, Opportunity, Regret, Productivity

Delphi (Method)
Perceptions and System Monitoring

Measures of user satisfaction, efficiency of use, quality and effectiveness of applications

Management/Operational practices
Performance of Systems/Technology

Sources of Knowledge and Wisdom IV

Sociology

Conceptions of technology
Social and organizational systems
Group processes and behavior
Environmental interaction

Information Science

Data base Technology
Cognitive use of data and information
Information Retrieval

Sources of Knowledge and Wisdom V

Anthropology

Understanding metaphors and culture (Method)
Cultural adaptation
Human learning process

Sources of Knowledge and Wisdom VI

Archaeology

"study of artifacts" (Method)
Tools and toolmaking

Philosophy

"virtual reality"
Design of Social Systems
Virtuality

Sources of Knowledge and Wisdom VII

Computer Science & Information Systems

Designer Wisdom
Expert Systems
Interface Design
Applications
Computer Mediated Communications
Systems Analysis & Requirements Development
Process Reengineering
Enterprise Design

DIMENSIONS OF INTERFACE DESIGN Subject two

CRITERIA FOR FACTOR DIMENSIONS

Can it be perceived?
Can it be measured?

Reproducible (repeatable)
Reliable (valid measure)

Orthogonal (Fundamental/Unique)
Can it guide design?

Relate to Interface design choices

Can it evaluate design?

Used to compare different designs

Can it sensitize designers?

Make them better designers

BASIC CATEGORIES OF DIMENSIONS

Foundation Factors
Understanding & Ease of Learning
User Sense of Control
Effectiveness
Psychological & Sociological
Administrative & Management

FOUNDATION FACTORS

Timeliness (Objective)
Responsiveness (Cognitive)
Reliability
Accessibility / Convince
Efficiency / Least Effort
Security
Accuracy
Protection/Bullet-proofing

UNDERSTANDING / EASE OF LEARNING I

Guidance / Informativeness
Conciseness / Brevity
Clarity / Simplicity
Comprehension
Segmentation / Decomposition

UNDERSTANDING / EASE OF LEARNING II

Consistency
Retention
Specificity
Familiarity
Scaffoling

SENSE OF CONTROL I

Leverage / Modifiability
Manipulability
Closure / Confirmation / Notification
Feedback / Sense of causality (Synchronous)
Contextual Visibility
Multi-tasking
Process Control / escape / interrupt / restart

SENSE OF CONTROL II

Forgiveness
Transparency
Flexibility / Cognitive Adaptation
Predictability / Regularity
Tracking / Monitoring

Backtracking / Auditing / History
Forecasting / Anticipating / Alerting
Backup / Undo / Listing

EFFECTIVENESS I

Task Functionality

Generality
Matching
Completeness
Abstraction / Leverage
Adaptability / Manipulability

EFFECTIVENESS II

Integration / Connectivity
Resiliency / Robustness / Richness
Relevance of objects and functions
Precision (minimize non relevant material)

PSYCHOLOGICAL & SOCIOLOGICAL I

Ethical / Honest / Private
Aesthetic / Pleasing / Artful
Interesting / Challenging / Fun
Self Image Enhancement
Peer Relations / Status
Sense of Community
Humanization / Polite

PSYCHOLOGICAL & SOCIOLOGICAL II

Satisfaction
Self, System, Group / Organization, Task / Job
Motivation
Expectations / "Self full filling prophecy"
Perceived Utility
Change & Social Inertia
Feeling of Participation

ADMINISTRATIVE

Training / Documentation
Maintenance of System
Job Enhancement
Human Help & Contact
Organization Relationships
System Evolution / Modifiability
Evaluation / Feedback
Costs and Charging
History & Confidence

EXAMPLES OF CONFLICTS AND TRADEOFFS

Comprehension <> Segmentation
Consistency <> Efficiency
Consistency <> Least Effort
Conciseness <> Informative
Conciseness <> Closure
Resiliency <> Ease of Learning
Task Generality <> Task Matching
Specificity <> Familiarity
Leverage <> Manipulability

COMMON DESIGN FAULTS I

Poor Interactive Sequences
Longer to do than manual (e.g. reading)
No tolerance or respect for human error
No flexible parsing (rigid syntax)
Wrong functionality
Start-stop hassle / Interruption handling

COMMON DESIGN FAULTS II

Integration across functions
Poor documentation and help
Inconsistent Metaphors
Lack of flexibility
Applicability to broad range of tasks

COMMON DESIGN FAULTS III

Multiple approaches to a given task
Multiple ways of invoking tasks
Alternative sequencing of functions
Adapt to different user styles / user types

INTERACTION METHODS I

MENUS: comprehension
COMMANDS: flexibility
LISTS: leverage
FORMS: clarity
DIALOGUE: guidance
WINDOWS: multi-tasking
ICONS: object representation
GUI: special representations
DIRECT MANIPULATION: control

INTERACTION METHODS II

MACROS: leverage
MIMICKING / RECORDING: task matching
ANIMATION: event sequencing
MODELS: abstraction
LANGUAGES: semantic representations
SCRIPTING: leverage, task matching
VIRTUAL REALITIES: understanding
AI AND EXPERT SYSTEMS: efficiency

THE DESIGN PROCESS Subject Three

DOING DESIGN

Any type of design component can be the stimulus for an idea
Design can occur in any order through the maze of component relationships
Any incident of a component can be the stimulus for the idea process
Design is a non linear thought process

DESIGN COMPONENTS I

Goals and Objectives
User Tasks & Requirements
System Metaphors

System Artifacts/tools

ISSUES

How to make the computer recipe system more valuable to the user than any of the regular physical systems he or she can use?
How to extend the concept of what is in a recipe to be more than what the user initially conceives it to be?

RECIPE SYSTEM OBJECTIVES I

The user should be able to handle thousands of recipes.
Provide ways of organizing the information that is superior to that provided by books and index card files.
Identify useful functionality that is easier to take advantage of on the computer.

RECIPE SYSTEM OBJECTIVES II Insure that individuals can learn to use this system very easily and be able to accomplish frequently occurring operations with a minimum of effort.

Provide facilities for the user to be able to learn more about cooking.

USER TASK: search requirements

Meals and meal parts (e.g. breakfast, snack, feast, main course, soup, desert, drinks, etc.)
Ingredients (meat, eggs, fruit, vegetables, etc.)
Nutritional (low fat, vegetarian, high protein, low salt, etc.)
Cultural (Chinese, various holidays, religious, ethnic, national, etc.)
Taste (hot, sour, sweet, bitter, coarse, fine, aromatic, fragrant, etc.)

RECIPE SYSTEM METAPHORS

Recipe, Meal
Cookbook
Card File
Scrap Book
Picture Metaphor

Dining Room, Kitchen, Root Cellar, Pantry, Menu (soup plate, etc.)

RECIPE TOOLS/ARTIFACTS

Nutritional analysis

e.g. calorie counter

Shopping list accumulation
User / Family Preferences Profile

e.g. Filters: likes, dislikes, necessary (low fat, diabetic, low salt)

Glossary, Historical Information
Food and meal layout function

TOOL EXAMPLE: RDA calibration Function to modify Recommended Daily Allowance (RDA) values for individual (lower salt, etc.) which would change nutritional calculation

Total Fat, Saturated fat, Polyunsaturated fat, Monounsaturated fat, Cholesterol, Sodium, Carbohydrates, Dietary Fiber, Protein, Sugars, Vitamins, etc.

Would require a master ingredient list giving all such information per unit of ingredient.

DESIGN COMPONENTS II

Objects / sub-objects

Object Parts
Representations

Shortest: used to shorten lists
Abstract: provide overview
Content: to fill in total object

RECIPE OBJECT PARTS

"title"
Descriptors for indexing
Auxiliary information: e.g. number of servings, side dishes, drinks
List of "ingredients" and the "amount" of each
Set of cooking "instructions"
User notes, markups
Modification alternatives for special occasions, added by user

RECIPE REPRESENTATIONS

Shortest: Unique title
Abstract: title, index terms, auxiliary data
Content: ingredients, instructions, links

OTHER POTENTIAL OBJECTS

Meals
Key word index
Ingredients index
Marked list

MEAL/MENU Object

Classical Meal Object

Appetizers, Drinks, Breads, Soup, Salad, Cleanser, Main course(s), Vegetables, Starch, Desert, After dinner drinks

User should be able to define meal structure for different types of meals

dinner, lunch, breakfast, brunch, family style, picnic, etc.

Compilation of all ingredients for shopping list

DESIGN COMPONENTS III

Functions on objects

Generic (detail depends on object)
Explicit (self defined)
Strategic choice sets
Reactive choice sets
Controls

GENERIC & EXPLICIT FUNCTIONS

Generic:

Get, Find, Create, Modify, Delete

Explicit Commands for Index

Add, Remove (term), Index (as verb) object, Change (term), Merge (terms), Split (term), Order (index), vote STRATEGIC MENU/CHOICES

Object	Number	Actions
Recipes	3,455	Get
Meals	238	Find
Lists		Changes
Keys	485	Create / Add
Ingredients	9,989	Modify /Change
Marked	35	Delete / Remove

EXAMPLE REACTIVE FUNCTIONS

Acting on some subpart of an object

Modify/add notes
Nutritional Analysis
Find Substitute/Alternative
Add to shopping list

REACTIVE MENU SAMPLE

Scan/List/View
Find
Mark/unmark
Add (index term/instruction/ingredient))
Remove (index term)
Controls: Home (escape), End (finish), Help, Show

EXAMPLES OF CONTROLS

Go (how far), back, forward, elsewhere, open/close/size windows, etc.
Print/File (how much): object, screen, page, list, window, link, etc.
Task control, confirm, quit, help, undo, escape, finish, interrupt, continue, copy, cut, paste, open, close, etc.

One can trace the evolution of interface design through the evolution as to what has become a commonly acceptable control "tool" DESIGN COMPONENTS IV

Modifiers and Status states

Subsets, tracking
creates meaningful lists

Lateral semantic linkages
Shared processes

List Processing
Searching

MODIFIERS

Create subset/list of objects
Examples

Authored recipes
Incomplete recipes
Sent messages
New messages

LINKAGES I

Meal to recipes
Recipe to meals
Ingredient

as recipe
to details
recipes/meals utilizing it
to substitutions

Number of servings to change program

LINKAGES II

To added Information

cooking method
utensils
nutritional details

Name to similar recipes
Related items: e.g. wine, side dishes
Time to prepare, to cooking

LINKAGE TYPES: Open Ended

Two way usually desirable

Part of (recipe in meal)
List generation (all meals with recipe)

More than one link from object/anchor

Alternative/substitution (of ingredient)
Explanation/Information (about ingredient)

Association/equality (other recipe)
Trigger program (e.g. servings, nutrition)

OBSERVATION ABOUT RECIPE

Natural contextual menu

user knows context
user can be taught context

User can learn easily through trial and error
Ease of learning clear

LIST PROCESSING

Adding and removing: index terms, indexed objects
Dividing, Merging, or Sorting lists
Comparing lists (multiple object entries)
Inverting (all occurrences)
Expanding, Contracting levels
Marking

KEY LIST SAMPLE I

meat

veal

...

beef

sweetbreads
brain
red meat
roasts
...

KEY LIST SAMPLE II

roasts

London Broil
Eye
Rump
...

steak

...

processed

hot dogs

ALTERNATIVE KEY STRUCTURES

Turkey

Holidays

Thanksgiving
Christmas

Leftovers

...
Fowl

Turkey

White meat

Breast

KEY LIST OPTIONS

Meat levels: 9 Items: 455
Use + or ... or "number" to expand or contract
Allow objects in multiple locations
meat.beef.red_meat.steaks.skirt (Shows where it came from)

SEARCH ISSUES I

What specific reactive searches are needed throughout the interface
Initiating the search and being informed of the results with respect to the amount of material found.

SEARCH ISSUES II

Deciding and indicating what is to be retained or discarded in a search result.

Whether to initiate a search of material found or to search the material not retrieved and how to combine this with the prior searches. DESIGN COMPONENTS V

Object Formats
Screen layouts

Workspace, status areas
Control areas, message areas

OBJECT FORMAT Rattlesnake Stew

Created: 8/12/91 Modified: 6/5/92
Keys: /meat.snake/exotic/rich/

American.southwest.Indian/

Servings: 2 Preparation Time: 90 min.

Last used: 12/8/92 Times used: 15

Characteristics: High Fat, Stewed

Utensils: Simmering Pot

Remarks: Supplied by John Franklin from my National Guard unit. Tastes very much like rich oily chicken.

INTEGRATION EXTENSIONS

Electronic Shopping
Maintaining inventory
Exchange recipes with other users
Group Meal Planning
Education: Virtual World Recipe System

Being able to actually replicate the cooking process in the virtual world.
Being able to have an instructor join you in that environment.
Being able to cook as a group.

DESIGN COMPONENTS VI

Processes and Closure

Processes and events that take place independent of user.

Interaction process & modes

The states and modes a user can be in.

User Object lists

The lists of objects the user will need or desire to manipulate.

DESIGN COMPONENTS VII

Error conditions

Identification of possible errors that the user could make.

System Messages

Notifications, Closures, Confirmations, and Error Messages.

Necessary Help

Identification of what is unclear or not self evident in the interface.

DESIGN COMPONENTS VIII

For Specialized Systems

CMC: roles, notifications
DATABASES: forms, entity models, reports
COMMUNICATIONS: protocol scripts/templates
HYPERTEXT: anchor, linkage types

Design Levels I

GENERAL	SPECIFIC	DETAIL
Goals	User Task
Metaphors
Objects	Heading, Abstract Content	Object parts Sub-objects Formats Linkages
Functions / Actions	Generic Links	Explicit, Controls

Design Levels 2

GENERAL	SPECIFIC	DETAIL
Strategic Choice Sets	Reactive Choice Sets
Shared Processes	Searching List Processing
Interaction States	Screen layouts Error Conditions Necessary Help	Workspace, Status, Control, & System message area

PROTOCOL ANALYSIS Subject Four

Language is the dress of thought

Samuel Johnson

WHY DO IT?

Objective: To discover the process a person goes through in solving a problem.
Uses: Learning Cognitive Processes, Developing Expert System material, evaluating interfaces.

KEY ASSUMPTION

Cognitive processes that generate verbalization are subset of those that generate behavior
Example: Lisa Learning, by Carroll and Mazur, IEEE Computer November 1986.

CONCERNS ABOUT THE METHOD

Subjects may have incomplete knowledge of their thinking processes; therefore the record may be incomplete.
Subjects may not have an accurate understanding of the processes of which they are aware.
The thinking process may be distorted by the thinking out loud process.
Ambiguity in language may lead to miscommunications.

NECESSARY ATMOSPHERE

Honesty
No evaluation of subject
No pressure for performance
No introduction of bias
No contamination of mental process (e.g. help)
Reciprocity and Respect

STIMULUS RESPONSE METHOD

Different stimulus may produce different mental behaviors

Do you know the capital of Sweden?
Which of the three: Oslo, Stockholm, or Copenhagen is the capital?
Name the capital of Sweden.

How much does it cost to fly and elephant from Thailand to Los Angeles?
How long is the Nile river?

MODES OF PROBING

Talk Aloud, Think Aloud: While information is attended.
Concurrent Probing: While in short term memory.
Retrospective Probing: After completion of the task.

PROTOCOL ANALYSIS CONDITIONS

Subject asked to verbalize what they are thinking
Subject is not being evaluated
Observer must not participate in process
Observer must not aid the subject
Subject providing knowledge of how they solve or a problem (or learn a system)

PROTOCOL ANALYSIS PROCESS I

Present the subject a written explanation of what is taking place.
Explain that this is to evaluate the system and not them
Explain you are there only to observe and can not help them.
Present in writing a task written the user terminology and the way the user would think about it.

PROTOCOL ANALYSIS PROCESS II

Total time should take between thirty minutes to an hour if no major problem encountered.
Observe and record (video, voice tape, PC interactions, notes and coding)
Only interrupt user for further verbalization if it is unclear what they are doing.

PROTOCOL ANALYSIS PROCESS III

Ask the user to describe what he or she is doing outloud.
Ask the user to go through all the terms on the screen and explain what they think they mean
Ask the user to forecast what they think an action will do
Carry out the task on the system

PROTOCOL ANALYSIS PROCESS IV

Give help only if user is at a dead end
Questions you can ask during if necessary

Why do you do/say that?
What is troubling you?
How do you know that ...?
Why do you do it this way?

Save retrospective questions for end of session.

MAJOR LIMITATION

Can not use it on a task that the user is not familiar with.
Requires training on the task first.

TYPICAL WRITTEN INSTRUCTIONS

PLEASE EXPLAIN:

What you think something means.
What you are trying to do.
What confusion or concerns you have.
What you expect to happen next.
What you don’t know the meaning of.

CODING SCHEMA FOR VERBALIZATION I

EXAMPLE I:

Intentions: goals, shall, will, must, have to
Cognitions: current attention situation, define, understand
Planning: If x than y
Evaluation: Yes, No, Damit, Fine
Changing conditions \ view of the problem

CODING SCHEMA FOR VERBALIZATION II

EXAMPLE II:

Surveying given information
Generating new information
Developing a hypothesis
Unsuccessful solutions
Self reference or criticism
Silence

PROTOCOL ANALYSIS OBSERVATIONS I

Verbalization occurs only 30% to 50% of the time.
Subjects cannot verbalize when:

Reading text
Doing intense cognitive activity
Making choices

Subjects have to slow down to verbalize
Subjects will tend to be more careful as a result of verbalization and observation

PROTOCOL ANALYSIS OBSERVATIONS II

Experts on a problem verbalize a lot more than non experts (double)
Experts have more difficulty verbalizing at a very detailed level with respect to the problem with which they are dealing with.
Experts usually spend more time in planning and in the restructuring of the problem.

INCREASING VERBALIZATION

1. Hold back stimulus or encourage slowness.
2. Segment stimulus (subtasks)
3. Interrupt with pre-arranged signal or set point to pause

e.g. when you are ready to indicate an action, first explain what you think everything means on the screen.

OBJECTIVES FOR INTERACTIVE SYSTEMS I

Determine their understanding of terms in the interface
Understand the cause of errors or misinterpretations
Determining missing functionality or user requirements

OBJECTIVES FOR INTERACTIVE SYSTEMS II

Determining reactions to and utility of features
Determine the utility of a metaphor for learning
Determining the utility of help and guidance

PROTOCOL ANALYSIS ADVANTAGES

A lot less effort than other approaches.
Can be done with prototype or mockup before any coding.
Learning how user approaches task
Finding major mistakes in design
Can learn attitude
Rapid feedback from small samples
Also useful for understanding user task

PROTOCOL ANALYSIS REQUIREMENTS

Subjects must be representative
Three subjects for each distinctive type of user and for each different set of tasks
Instructions simple
Observe only

PROTOCOL ANALYSIS QUESTIONS

Can ask/say:

Please explain your choice.
What are you thinking?
What does that term mean?

Should not ask:

Why did you do that?
What does "append" do?

Have physical signal for interrupt

PROTOCOL ANALYSIS: HOW TO I

A one page explanation to the subject
Set of written tasks in user terms
Subject should only spend 40-60 minutes.
Categorization scheme for recording
Tape record their verbalizations

PROTOCOL ANALYSIS HOW TO II

Retrospective questionnaire for end
Retention of major concepts
Perceived utility of features/functionality
Do not try to test everything
At least three subjects on same task
Be specific about user explaining choice they are about to make

UTILITY OF MOCK UP

Ease of understanding (current knowledge)
Ambiguity in terms
Confusion generation
Loss of continuity
Developing on line help
Marketing and acceptance

UTILITY OF WORKING SYSTEM

Ease of learning (new knowledge)
Error impact & Utility of feedback
Ease of exploration
Realistic task execution
Developing final user documentation
Utility of new features (beyond current system)

Wizard of Oz System

Allows simulation of working system
Dummy output data stored
Working choices for moving among screens and windows
More software to do this appearing on the market

BASIC QUESTIONS I

Do the terms used on the screen mean to the subject what the designer thought they would mean?
Do the alternatives presented at that point in the interaction include what the subject wishes to do?
Is the help material or the system messages understandable or relevant to the needs of the user?

BASIC QUESTIONS II

Does the subject have difficulty locating or perceiving things on the particular screen?
Does the subject utilize the sequences of operations that the designer expected to be utilized in accomplishing a given task?
Can the user utilize the interface metaphor for learning the system?
What type of errors is the user making and why?

WHEN TO USE

Protocol analysis should be used
Before system requirements are finalized
After every major change to requirements or interface
Before instillation of system for users
Before introduction to new user population type

INDEXING & SEARCHING Subject Five

"When I use a word," Humpty-Dumpty said, "it means

just what I choose it to mean -- neither more nor less."

--Lewis Carroll, Alice in Wonderland

PART OF THE PROBLEM Bible, Holy Bible, La Sainte Bible, Biblis Sacra, Bible wordo, biblia, Scriptures, Holy Scriptures, The Scriptures, Sacra Scrittur, Saintes, New Testament, Old Testament, Testament, Nouveau Testament, Avrcien, Vetus Testamentum, vetus novum, nuovo testament, gospels, evangelium secundum, mathhewm, gospel of St. Matthew, epistle to the romans, acts of the apostles, proverbs, psalms ecclesistes. SEARCH MATCHING PROBLEM

IF DEALING WITH COLLABORATIVE INPUTS TO DATA

Air Force Base
air force base
AFB
A.F.B.
A. F. B.

MULTIPLE MEANINGS: AMBIGUITY

What is a "tank"?

weapon
oil storage tank
fish tank
part of toilet unit

May need context to resolve
Use of jargon and codes in groups

INDEXING TYPES

Hierarchical
Network (lateral)
Subject headings
Key words
Syntactic languages
Phrases
Natural Language

HIERARCHICAL

Single location in tree
Precise object (definition of item)
e.g. 1, 1.1, 1.2, 2.1, 2.1.1
e.g. outlines, MDSE. files
Minimum coding size, difficult to do and use
Rigid, difficult to evolve / adapt

NETWORK (LATERAL LINKS)

Single Location in network
Precise relationship (definition of link)
e.g. book index, citation index
e.g. Hypertext (in theory, not practice)
Lack of global view

CITATION INDEX I

Index specific to a time like last year or last five years
Source of data all professional papers published during that time frame (on line version for all years recorded).
Papers published in refereed journals and conferences

CITATION INDEX II For a paper published at any time in the past and referenced by papers published in that time frame, the results is a list of which papers referenced the earlier papers.

Uses reference list at back of papers to create an "inverted index"

CITATION INDEX PROPERTIES

Most papers cease to be referenced five to ten years after publication
Most important contributions to science are papers reference ten and twenty years after publication
Some fields have significant number of papers never referenced (e.g. math is 30%)

CITATION INDEX UTILITY

Useful for finding recent work in a given field defined by important papers in the field.
Useful for evaluating the contribution of individual scientists over the long term.
Takes 3 to 5 years after publication to begin to assess importance of individual paper.

SUBJECT HEADINGS

Fixed categories, no structure, categorical scale
Precise definitions of bin in which to fit items: Problem: book on "Use of integral equations in Physics"

Open choice as to file objects in multiple headings but not commonly done except through lateral linkages (e.g. cross references) KEY WORD AND COORDINATE SYSTEMS

Fixed key words
Free key words
Multiple keys for single object
Coordinates for properties

e.g. tall, medium, or short

SYNTACTIC LANGUAGES: Tagged Descriptors

Qualified keys, associated subcategories
Mix of key word with attached hierarchies

e.g. tank.weapon or tank.petroleum
e.g. steak.beef, steak.lamb, steak.veal

(type of meet).(source)

SYNTACTIC LANGUAGES: Faceted Index

Separate dimensions to represent item

Like the many facets of a diamond

softness, style, color, grade
reference: author, title, source

Very common in science and commerce

leather, wine, metal alloys, plastics

Both objective and subjective dimensions

MIXED INDEX EXAMPLE

UNIVERSAL DECIMAL CLASSIFICATION

341.67:623.454.8(094.2)

341.67: Disarmament, limitation and control of weapons of mass destruction
623.45: Ammunition, pyrotechnic devices, weapons of mass destruction
623.454.8: Active rays, atomic nuclear (thermo) weapons
094.2: Historical sources (09), International treaties
Why is nuclear weapons a subcategory of ammunition?

NATURAL LANGUAGE: Phrases

Short phrases, titles

chapter/book titles, paper titles
e.g. KWIC index

Often poor because done by original author

KWIC INDEX

Key Word In Context
Alphabetical listing of all titles permuted word by word
Words like of, the, a, an etc. not used.

Acceptance of Groupware Systems.
Groupware Systems. Acceptance of
Systems. Acceptance of Groupware

NATURAL LANGUAGE: Abstracts, Paragraphs

Long enough to distinguish the single item uniquely from all others in the collection of items

Good when searchers and writers of abstracts are experts in domain
English ambiguity a major problem

INDEXING EFFECTIVENESS I Relevant Non Relevant

Retrieved A B

Not Retrieved C D

Precision = A / (A+B)

Recall = A / (A+C)

Specificity = D / (B+D)

Total Size Dependent (D)

Search Efficiency = (Recall)x(Specificity)

INDEXING EFFECTIVENESS II Cannot have both high precision and high recall with qualitative information and even very complex quantitative databases (e.g. all known plastics, metal alloys, etc.)

Indexing methods represent compromise between these two opposing objectives.

INDEXING EFFECTIVENESS III

Too much recall leads to information overload and/or more effort for user
Too much precision leads to loss of relevant material and poor quality result for user

INDEXING EFFECTIVENESS IV

Timeliness (updating and currency)
Accuracy
Completeness (All data in database)
Form/structure of data useful (e.g. summary, raw data)
Depth and Breath (depth easier for novices, breath better for experts)

INDEXING EFFECTIVENESS V

Discrimination & Resolution / Density
Subjective / Objective
Adoption & Acceptance
Historical relevance
Integration (analysis tools, other sources, etc.)

EVALUATION DIMENSIONS I INDEX TYPE AMBIGUITY EXPRESSIVE CONCISE HIERARCHICAL
LOW LOW HIGH
NETWORK
SUBJECTS
FIXED KEYS
FREE KEYS
TAGGED DESCRIPTORS
FACETED INDEXES
PHRASES
NATURAL LANGUAGE

HIGH HIGH LOW

EVALUATION DIMENSIONS II INDEX TYPE                     RETRIEVAL                 SELECTION                 ADOPTION                                      EFFORT                       EFFORT                     EFFORT
HIERARCHICAL
                                       LOW                             HIGH                             HIGH
NETWORK
SUBJECTS
FIXED KEYS
FREE KEYS LOW
TAGGED DESCRIPTORS
FACETED INDEXES
PHRASES
NATURAL LANGUAGE
                                     HIGH                                 LOW                             HIGH IDEAL USE OF INDEX TYPES I

HIERARCHICAL

Macro concepts, well structured area (table of contents)

NETWORK

Micro concepts, structured relationships (hypertext document)

SUBJECTS

Macro concepts, structured objects (professional papers)

IDEAL USE OF INDEX TYPES II

FIXED & FREE KEYS

Micro concepts, unstructured, evolving (book index, files)
Free if area is evolving
Fixed when creators know more than users

IDEAL USE OF INDEX TYPES III

TAGGED DESCRIPTORS & FACETED INDEXES

Micro, structured factors, well understood dimensions (e.g. materials, commodities)

PHRASES

Macro, semi-structured (message titles, paper titles, book titles, etc.)

NATURAL LANGUAGE

Macro, unstructured (abstracts)

ZIPF’S LAW

Log frequency of terms in the English language (Y axis) are linear with log of rank order (X axis).
Product of frequency and rank order raised to fixed power is a statistical constant
Non log plot often called "draw down curve" and used in other fields.

SAMPLE Zipf List I Word Rank Frequency R x F

the            1         68,315         68,315
of              2         35,716         71,432
and           3         27,856         83,568
to              4         26,760        107,040
a                5         22,744        113,720
in              6         21,108         126,648
that           7         11,188         78,316
is               8         10,978         87,824
was           9         10,499          94,491
it              10        10,010         100,100

Sample Zipf List II Word Rank Frequency R x F

sir           195             452         88,140
it's          196             452         88,592
why        197             451         88,847
asked     198             448         88,704
give        199             446         88,754
once       200             443         88,600

Sample Zipf list III Word Rank Frequency R x F

usually 400             239         95,600
tax         500             167         83,500
ideas      800             128         102,400
proved   1,170          88           102,960
sections 2,146         49            105,154
flames 5,070            17             86,190
cultures 7,020         11             77,220

OBSERVATIONS ON LANGUAGE

Order: Easiest to say to hardest to say (amazing collective evolution of language)

Can be derived from Classical Information Theory by minimizing total length of codes to represent message. Done years later after Zipf’s empirical studies. CONSEQUENCES

Led to Zipf’s Principle of Least Human Effort
Human Behavior and the principle of least human effort: an introduction to human ecology, George Kngsley Zipf, 1949, 1972.
Concepts extended to many other areas of human and biological activity.

SEMANTIC BALANCE I

Speaker’s economy in possessing a vocabulary of one word with m different meanings.
Auditor’s (listener) economy in possessing a vocabulary of m different words with one distinctive meaning.
Conflict in language and cause of "ambiguity." A collection of m words will have more than m meanings.

SEMANTIC BALANCE II First 1,000 words (after initial 500) have like 6,000 meanings, meaning they are used much more often than later words. (Ave. = 6)

First 20,000 words have like 40,000 meanings (Ave. = 2)
Suitable packet (unit of words) size can fit harmonic series
Compromise and tradeoffs in human affairs

Zipf RELATIONSHIPS

Form of equation from: Generalized Harmonic Series
r x f ^ (1/p) = C
r = rank f = frequency
1 >= p >=0
for English p ~ 1, but specialized vocabularies differ slightly.
Distribution of star types provides insight into evolution

ZIPF’s STUDIES I

Number of retail establishments vs. population
Diversity (number of different establishments) vc. population
Vocabulary of NY Times vs. ordinary paper (constant a lot larger)
Tool layout for craftsperson (furniture makers, shoe makers, watchmakers, etc.)

ZIPF’s STUDIES II

Frequency of marriage vs. distance in blocks (before cars)
Visitors to national parks vs. distance of home
Size of cities (derived from agricultural model) before last few decade
Number and diversity of establishments vs. size of population
Biomass (frequency x weight) in pond (closed ecological system) constant among species in food chain.

SYSTEMATIC KEY WORD LIST DEVELOPMENT I

Analyze body of literature (significant sample) in application domain
Compare order to normal English and highlight significant differences

Concordance study useful for any writers

High frequency useless: too many matches

SYSTEMATIC KEY WORD LIST DEVELOPMENT II

Low frequency useless: too few matches
Use middle frequency terms different from normal English

Do correlation studies of two, three, four, and five terms together (within same paragraph, sentence) to seek clusters of matches. PREPARING THE SAMPLE

Remove useless words ("Exclusion list) from initial sample
Illustration of "a" words from such a list:

a about above after afterall afterward again against all almost already also although always am an and another any anyone anything are as at

Remove suffixes: (e.g. age als ant ary) from all words in sample

SEARCH APPROACHES I

Scanning: Broad, top level, term lists
Browsing: Up down, laterally, following interest, iterative, backup and start over
Target Searching: Specific target objective, need structure of data, categorization rules and understanding

SEARCH APPROACHES II

Exploring: Objective is learning the nature of the database and contents.
Wandering: Creative stimulus, random patterns

Both are a significant design challenge

SUBJECTIVE HUMAN INDEXING

Spatial Indexing (e.g. piles of paper)
Event Indexing (e.g. calendars)
Color & Form Indexing (e.g. tabs, marks)
Categorical (e.g. priorities)

SEARCH PROCEDURES I 1 Perceiving a need to find

information.

2 Formulating the request for

information.

3 Selection of the source for the

information.

SEARCH PROCEDURES II 4 Specifying the search strategy.

4.1 Browse the index, provide

numbers

4.2 Expand and contract levels

4.3 Mark multiple choices (or)

4.4 Provide for: or (novice), and, not

conditions, also distance

conditions

SEARCH PROCEDURES III 5 Carrying out the search

(mechanics).

5.1 Show number of hits

dynamically

5.2 Browse the hit list

5.3 Mark, Nest, Backup, Merge,

Restart

SEARCH PROCEDURES IV 6 Evaluating the results

6.1 Update index or list. Determine

relevance

6.2 Saving, printing, viewing,

categorization

6.3 Merging, reordering, clean up,

store strategy

End of Lecture 1-5 Set