Syllabus -- CIS 675 EVALUATION OF INFORMATION SYSTEMS

(Section 852)
(last modified: December 2005)

New Jersey Institute of Technology
Spring 2006-Prof. Roxanne Hiltz
email: Hiltz@njit.edu

http://web.njit.edu/~hiltz (course notes posted there to print, via streaming video at CPE site)

Office phone: 973-596-3388. Office location, 4104 GITC, corner of Central and Lock. Hours on Mondays are scheduled for 11:30 - 12:45, 4:15 - 5:45, and by appointment from 6 pm until 8 pm. Email me for possible appointments on other days or times; I am usually on campus and available by appointment on Wednesdays. See my calenda http://calendar.yahoo.com/roxanne_hiltz, look for a day and time when it says "on campus and available for appointments" that is not taken yet.

COURSE OBJECTIVES

1. To give you enough background about information systems theory and research methods so that you can critically read and understand the research articles that appear in leading IS journals such as MISQ, JMIS, Information Systems Research, ACM publications, and the human-computer interaction journals. This will enable you to keep up in the field after you graduate, as well as to carry out the first step of any future research project, the "literature search."

2. To give you enough skills in information systems research techniques to be able to carry out an applied project to design and implement an evaluation plan that will improve the effectiveness of an information system in an organization. In combination with at least one additional course in multi-variate analysis, this course also provides the foundation for doing publishable research on information systems.

Required readings and materials:
Rosenthal, Robert, and Ralph, Essentials of Behavioral Research: Methods and Data Analysis, McGraw-Hill, second edition, 1991. (Note; 3rd edtion is in press but not quite ready; I plan to put a photocopy of the Ms. on reserve in the library).

Lectures:
Available in Powerpoint plus audio via CD-ROM through CPE; note that several are newly revised for Spring 2006.
Journal articles listed below are available online through the NJIT Digital Library collection, or through the URLs listed.
Some selections are marked "optional:" masters students need not read these articles (though to the extent they are covered in lecture, they are responsible for what is in the lectures); but Ph.D. students need to read them for possible use in their qualifying examination. Some are marked "skim:" this means just take a quick look at it and try to get the main ideas. If by any chance a 475 student is in a 675 section, he or she need not read the "optional" or "skim" selections.

All students must have an account on WEBBOARD as part of this course, and must be able to access the world wide web via a browser. YOU ARE EXPECTED TO PARTICIPATE IN THE CLASS DISCUSSIONS AND ACTIVITIES ONLINE, EVERY WEEK. THIS WILL CONSTITUTE 15% OF YOUR GRADE. For this purpose, the grading week will run from Sunday morning through Saturday night. Some weeks there will be specific activities (such as small group discussions with reporting back) and other weeks there will be only the ongoing assignment to make comments and ask questions about the readings and the lectures.

AN OPTIONAL BUT RECOMMENDED SYNCHRONOUS CHAT SESSION WILL KICK OFF THE COURSE ON TUESDAY JANUARY 18 FROM 8 - 9 PM EST. We will have a conference on WebBoard called "Synchronous Chat" where we will "meet." You may ask questions about any aspect of the course.

Note on "weeks" - "Week one" will be the first week of the spring semester, Jan 16- 22, even though Monday is a face to face holiday; Week 2 starts Monday Jan 23. All students are expected to have signed online and begun participation by Tuesday Jan 17; the WebBoard will be up and available well before that.

Note 2: This course requires approximately 10-15 hours a week of work, depending on your reading speed, including the time to listen to the lectures.


EQUIRED READINGS

I. Evaluating Information Systems: Theoretical and Methodological Foundations (2 weeks)

Topics: Philosophical perspectives, Concepts, Theories, Ethics

Text Reading: Rosenthal and Rosnow; Skim chapter 1, read carefully 2 &3
Skim: ISWorld site on Information Systems Effectiveness at http://business.clemson.edu/ISE/
Goodwin, Nancy C. , "Functionality and Usability," CACM March 1987, pages 229-233.
Keen, Peter, "Information Systems and Organizational Change," Communications of the ACM, January 1981, Volume 24, Number 1, pages 24-33.
Markus, M. Lynne, "Power, Politics, and MIS Implementation," Communications of the ACM, June 1983, Volume 26, Number 6, pages 430-444.
(Optional: See also: http://www.people.vcu.edu/~aslee/ICIS2000.htm, "Three Perspectives: If Markus's 1983 Classic Study Were Being Reviewed Today," panel at ICIS 2000)
Skim: DeSanctis, G. and Gallupe, B. "A Foundation for the Study of Group Decision Support Systems," Management Science, 33 (1987), 589-609 (note: an example of conceptual structure or typology)
Theoretical models (the example of TAM): Venkatesh, V. "Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model," Information Systems Research, 11,4 (Dec 2000)?for now read Introduction and Theoretical Framework, pages 342- 352 only.

II. Qualitative Research Methods: A Brief Overview (2 weeks)

Topics: Validity and reliability, Protocol Analysis, Case and Field studies, semi-structured interviews

Week 3 readings:
Text Reading: Rosenthal and Rosnow; skim 8, read 11.
Levi, M. and Conrad, F., A Heuristic Evaluation of a world wide web prototype, ACM Interactions, Jul- Aug. 1996. [ACMDL]
Boren, M. Ted and Ramey, Judith. Thinking Aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43, 3, Sept. 2000, 261- 278.
(Optional- all you need to know will be covered in lecture..) Boudreau, M.C., Gefen, D. and Straub, D.W. "Validation in information systems research: A state-of-the-art assessment. MIS Quarterly March 2001, 1- 16.
Skim: Holzinger, A. Usability engineering methods for software developers. Communications of the ACM 48, 1 (January 2005), 71- 74..

Week 4 (Feb 6) readings:
Rosenthal & Rosnow, chapt. 9
Skim: MISQ Discovery (online hypertext journal)? "Qualitative Research in Information Systems" http://www.auckland.ac.nz/msis/isworld/index.html
Vandenbosch, B. and Huff, S. "Searching and Scanning: How Executives obtain information from executive information systems." MIS Quarterly, March 1997, 81- 105 (be sure to look at the appendices).
Wanda J. Orlikowski. "CASE Tools as Organizational Change: Investigating Incremental and Radical Changes in Systems Development." MISQ, Vol 17, No. 3, September, 1993 . Accessible at http://www.misq.org/archivist/bestpaper/misq93.html
Skim: Myers, M.D. "Investigating information systems with ethnographic research," Communications of the Association for Information Systems, December 1999. (an online journal; this article is accessible from the Qualitative Research page at the following URL: http://www.auckland.ac.nz/msis/isworld/Myers%20CAIS%20article.pdf

FIRST EXAM, 2 HOURS, 10% OF GRADE, IS SCHEDULED FOR (week 5) MONDAY FEBRUARY 13, 6 pm, AND WILL COVER THE TOPICS AND READINGS ABOVE. I will proctor it in my office at NJIT (GITC 4104 ) or if you are more than 50 miles away you may arrange for a proctor.

III. QUANTITATIVE METHODS

A. Scaling and Structured Questionnaires: Sampling (Weeks 5-6)

Rosenthal & Rosnow, chapt 10 (skim)
Venkatesh, V. "Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model," Information Systems Research, 11,4 (Dec 2000) - for now read "Method" (pp. 352- 354) and Appendix 2 (pp. 360- 361) only.
Optional: Templeton, G.F., Lewis, BR., and Snyder, C.A. "Development of a measure for the Organizational Learning Construct," JMIS, 19,, 2 (Fall 2002), 175- 218.
Optional: Stanton, J.M., Snar, E.F., Galzer, W.K. and Smith, P.C. "Issues and strategies for reducing the length of self-report scales," Personnel Psychology, vol 55 (2002), pp. 167- 194.
SKIM: Survey Instruments in IS, by Peter Newsted, Sid Huff & Malcolm Munro In online MISQ Discovery: http://www.acs.ucalgary.ca/~newsted/surveys.html

(assignment 1 due week 6)

B. Methodological Articles: Statistical methods and Data Analysis (Weeks 7 and 8)

Rosenthal & Rosnow, 13 (skim); 14, 15, 23, 24 (skim)
Venkatesh, V. "Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model," Information Systems Research, 11,4 (Dec 2000) -¡°results¡± and rest of article (354- 365).
A.L. Lederer et. al., "The role of ease of use, usefulness and attitude in the prediction of world wide web usage." ACM CPR 98, 195-204.
Barki & Hartwick, "Measuring User Participation, User Involvement, and User Attitude," MIS Quarterly, March 1994, 59-82.
Optional: Barua et al, "An empirical investigation of net-enabled business value," MIS Quarterly, December 2004, 585- 620 ( an example of a study that did "all the right things" to achieve significant results and get published.)

C. Experimental Methods (Weeks 9 to 11)

Rosenthal and Rosnow, Chapters 4 (read), 5 & 6 (skim) , 16 (skim) 17 (read), 18 &, 19 (skim)
Connolly, Jessup and Valacich, "Effects of Anonymity and Evaluative Tone on Idea Generation in Computer-Mediated Groups," Management Science, June 1990.
Ellen Hoadley, "Investigating the effects of color," CACM, 33, 2 (February 1990), 120-125.
Dickson, DeSanctis, and McBride , "Understanding the Effectiveness of Computer Graphics for Decision Support," CACM, January 1986, 40- 47. (optional- but will be covered in lecture)
N. Tractinsky & J. Meyer,"Chartjunk or Goldgraph?" MIS Quarterly, Sept. 1999, 397- 420)
(Optional)- Lim, K.H. and Benbasat, I. "The effect of multimedia on perceived equivocality and perceived usefulness of information systems." MISQ, Sept. 2000, 449- 471.

Week 12 : ( EXAM 2 April 10 6 pm -- weeks 5-11 readings and lectures; 3 hours, 15% of grade)

IV. Designing a Complete Evaluation Plan (lecture 11/ week 12)

Alavi, "An Assessment of the Prototyping Approach to Information Systems Development," CACM, June 1984, 556-563.
Mantei and Teorey, "Cost Benefit analysis for incorporating human factors in the software lifecycle," CACM, April 1988, 428-439.

V. Selected additional methods and studies (lectures/ weeks 13 and 14)

Schmidt, Lyytinen, Keil & Cule, "Identifying Software Project Risks: An International Delphi Study." JMIS, Spring 2001
Saleem, N., "An empirical test of the contingency approach to user participation in information systems development." J. of Management Information systems, 13, 1 (summer 1996), 145-166.
(optional) Chang, Jerry Cha-Jan and King, W.R. (2005). "Measuring the Performance of Information Systems: A Functional Scorecard." JMIS, 22 (1) Summer 2005, 85-116.
(optional) Venkatesh, V., Morris, M.G., Davis, G.B. and Davis, F.D., "User Acceptance of Information Technology: toward a Unified View." MIS Quarterly, 27, 3 (Sept. 2003), 425- 478.

Week 15 (Monday May 1) : Project presentations, projects due on reading day two days later (May 3). All students are required to prepare a 5 minute presentation on their project, with Powerpoint Slides, and to either present them "face to face" - we will reserve a lab for this special meeting- ) and participate in the entire presentation session; (this will be 10% of the project grade) or to post the presentation online. Students who can are encouraged to come to the face to face presentations, where we will also have a "pot luck party." In addition, if you present the overview of your project and somebody notes something very "wrong," you will have a couple of days to fix it before the full project is due.

ASSIGNMENTS AND GRADING
(subject to modification)

Exams: 25% (10% exam 1 and 15% exam 2)
Final Project: 25%
Assignment 1 (Protocol analysis) 20%
Assignment 2 (Experiment design -) 10%
Class participation 20%
NOTE: All students are expected to contribute at least one comment a week to discussions of readings and lectures or other questions or issues raised for discussion.

Final detailed instructions and due dates for the following will be distributed online:

1. Week 1 participation (from about January 14 through January 21)
a. Get a WEBBOARD userid and join our 675 board.
b. Respond to my request to introduce yourself in the 675 Board's -Welcome conference. )
c. Reply to at least one other person's introductions to say hello and get to know each other, and to answer questions about logistics or systems issues.
d. Optional: participate in a synchronous chat about the course, January 17, 8-9 pm EST.

ASSIGNMENTS

Note: the assignments are usually done in groups. You will generally be preparing for the next assignment ( eg, forming groups and choosing a topic) while completing the prior one.

Assignment 1 Protocol analysis (20%) (Using a qualitative method to observe users¡¯ problems with a prototype, to find out how to improve the usability of a final version of the software). Instructions will be posted online. This will be done in teams and due by week . You will start choosing teams for this assignment the week ofjanuary 23, and start the actual assignment the week of January 30. The final reports will be due Feb. 27

Assignment 2 (15%)

Experimental design and participation; and/or article review and experimental design by you (depending on whether or not you participate in the experiment made available). Detailed instructions about the various options will be posted. This assignment will begin week 7 (Feb 27) and be due by March 20 (tentative, depending on scheduling of the experiment).

Exams: will cover all lecture material, required readings (no detailed questions on chapters marked "skim," or "optional" but you may choose to use them to answer essay questions); and online electures and class contributions by others.

Final project: Design an evaluation research plan for an actual information system in an organization, including a literature review of previously published related work, the methods and instrumentation you would use, to include at least one qualitative and one quantitative approach, and a cost-benefit analysis of the research itself. You need to show "sample/ hypothetical" results that would support your hypotheses, using the numbers from the sample size you choose. (You will not actually carry out this research for this course, though many students in the past have subsequently done so, in an organization in which they work.) Full text will be due (delivered to me at school or at home) on "reading day", in week 15 on Wednesday May 3.

GRADING:

General Policy: You will be penalized 2 points a day, or ten points a week, for unexcused latenesses. Some assignments need to be completed by certain dates to receive credit (e.g., if you are in an experiment, you must complete by the due date). No late assignments may be turned in after the last day of the class for credit. It may be possible to re-do an assignment for a better grade; ask; but in no case will a redone assignment receive a grade higher than 90, and such "redos" must be completed within an agreed upon time before the end of the course..

Students who due to exceptional circumstances find themselves unable to complete projects by the due date, or who will need an Incomplete in the course, should contact the instructor by private email or during office hours IN ADVANCE OF THE DUE DATE to negotiate what will be done. We will try to work something out. Having a baby in the middle of the semester or being posted to an active war zone are examples of such exceptional circumstances for which extensions have been granted in the past. If you will have a trip for your job, then you should complete assignments that will be due while you are gone, before you leave on your scheduled trip.

COPYING/CHEATING

Students are encouraged to share knowledge and experience as part of the learning process. However when actually producing an individual assignment, students are expected to produce that assignment from their own efforts. Presenting an assignment, or any part of an assignment, as your own work for this course when it is not is called cheating. So is using unauthorized notes during an exam, of course. Cheating goes against the principles of this institute, and is viewed very seriously; students who cheat do not belong in the Information Systems program at NJIT.

It is an important academic ethic that when quoting or using another author's writing, the source is recognized by a citation ( author, date) as well as listed in the references. If you are quoting directly, the quoted section must be in quotation marks and the page must also be cited. Not to do so is called plagiarism. Any evidence of cheating in any form, including plagiarism, will be dealt with according to the honor code of NJIT (course failure and suspension or expulsion; in one case known by the instructor, the student was expelled even though all other degree requirements had been completed). If you have any question about what constitutes cheating, discuss it with the instructor before handing in the questoinable piece of work, rather than taking a chance of ruining your academic life.

Questions regarding any assignment should be asked and answered in general terms (e.g."What do you think of media richness theory?"). In this way your classmates and others are "resources," and this is encouraged. However direct support for particular assignments (e.g. "can I copy your part 3 of the assignment?") should neither be requested nor given ? the requestor must be told to do their own work. Where it is clear that two or more assignments are largely the same, it is likely the assignment will be given zero marks for both students.

USING "TURNITIN.COM"

A copy of all individual assignments, including the final project, will be required to be submitted to the "turnitin.com" web site. This is in addition to paper copies that may be supplied to the professor, or to posting a copy of the assignment for the class, according to detailed instructions that will be given for assignments. This service checks your work against everything available on the world wide web and in the archives for all sections of 675 past and present, and gives a report on exactly where your work is similar to that of other work that can be found. This is an objective way to help determine whether plagiarism has occurred.

REFERENCES AND CITATIONS

Use standard APA citation format and references, as follows.
For a journal article:
Spark, J. S., Glow, J. P. and Twinkle, L., (1994). APA format for journal articles. Management Science, 28(10), 1187-1197.
For a book:
Spark, J. S., Glow, J. P. and Twinkle, L., (1994), APA Format For Books, New York: McGraw Hill, Ed. 3, 204-230.
In the text use "(author, date)", then list all citations in alphabetical order by author at the end of the assignment.