Division of Information Systems
Department of Computer and Information Science
New Jersey Institute of Technology
The paper describes the experience of trying to use Webboard to manage 40-50 active students in a single course which has high pragmatic content and encourages a collaborative approach to education. It presents the methods used to organize the course and both the advantages this CMC (Computer Mediated Communication) system offers and the problems it creates. Finally some proposals are made for the functionality need in CMC systems to make it easier to manage large classes.
Class size, Asynchronous Learning Networks, Distance Learning, Management of Information Systems
This paper is taken, in part, from:
Effectively Managing Large Enrollment Courses: A Case Study, Murray Turoff and Starr Roxanne Hiltz, CIS Department, New Jersey Institute of Technology, Sloan ALN Workshop 2000 (September Lake George, NY)
To be published JALN (Asynchronous Learning Networks), http://www.aln.org
This reduced version will be available on my website http://eies.njit.edu/~turoff
This research was partially supported by grants from the Alfred P. Sloan Foundation and the New Jersey Center for Multimedia Research. Student research assistants Chris Rodriguez, Bijal Desai, and Michael DellaVecchia assisted with the data gathering and analysis for this and other papers.
ABOUT THE AUTHOR
Murray Turoff is acting Chairperson and Distinguished Professor in the Division of Information Systems, Department of Computer and Information Science, New Jersey Institute of Technology, Newark NJ 07102. He can be reached at Turoff@NJIT.edu; 973-596-3366; and at his homepage is HTTP://eies.njit.edu/~turoff.
The project from which the data and case study presented here is drawn was called “From Virtual Classroom to Virtual University: Institutionalizing Asynchronous Learning Networks (ALN) at NJIT.” Previously (1993-1996), we had produced and evaluated the ALN offering of all of the major courses for undergraduate degrees in Information Systems and Computer Science. Analyses of effectiveness based on these data were presented in a previous paper (Hiltz et. al., 1999). The most important goal of this most recently completed project was to “institutionalize” ALN at NJIT; that is, to provide an infrastructure and set of programs that would be able to continue to function once the six years of financial support and faculty leadership provided by the Sloan Foundation grants ended in December of 1999.
At NJIT we have been employing group communication software to deliver distance education and to augment face to face classes since the early eighties. Our approach has always been oriented to employing collaborative education methods and we have also conducted extensive evaluation studies both of a field trial and an experimental nature. Most of that evaluation effort has produced strong evidence that distance education employing collaborative approaches can produce educational results as good as face to face and in some cases can even produce better results than face to face.
A significant factor in accomplishing this result is the nature of the software that provides the group communication process. For a long time we were able to use EIES (Electronic Information Exchange System) that we build and evolved since 1974 at NJIT. EIES was officially retired the summer of 2000. We now have three separate commercial systems that NJIT faculty may choose from to deliver distance education and/or augment face to face classes. Another significant factor is the facilitation and leadership role of the instructor. Since the ability to carry out that role as part of a group oriented communication process is also tied into the functionality of the system being used we will be explain the exact procedure we used to deliver a course and the constraints placed on the delivery by the software used.
The prior paper on this project described faculty training and motivation, details of course delivery, etc. (Hiltz et.al., 2000). In the following sections, we will look at the characteristics of the ALN students, as they compare to students in on-campus sections of the same set of courses, and at some of the results derived from the post-course questionnaire.
CHARACTERISTICS OF THE STUDENTS
A course survey was administered to students enrolled in ALN sections of courses in the Sloan ALN project. The ALN sections used video tapes (or in a few cases, CD ROMS or web pages) to deliver the “lecture” type material in a course, and Virtual ClassroomÒ on EIES at NJIT (or, towards the end, the WebBoard or Virtual University conferencing systems) to support class discussions and collaborative assignments. “Mixed mode” sections combined ALN with face to face meetings. We also surveyed students in a limited number of “comparison” sections. These were the same courses, taught by the same instructor or set of instructors, taught in traditional on-campus “face-to-face” sections. Instructors in the target sections were requested to hand out the questionnaire at the final exam, if there was one; otherwise, the questionnaire was mailed. For many either there was no on campus final exam, or the instructor did not cooperate in handing out the questionnaire or urging its completion.
Characteristics of the students in ALN distance modes vs. other modes are described in a series of cross tabulation tables which appear in at the end of this paper in the Data Appendix. As in the previous project, it was found that ALN students were significantly more likely than students attending courses offered on campus to be U.S. citizens (69%) and have English as their native language (60%); belong to an ethnic group considered “white” (51%); be distance students (40%) rather than commuters (47%) or resident on campus (13%); be married (40%); have children (29%); work 30 or more hours a week (60%); have their tuition reimbursed (48%); be non-matriculated in a degree program (10%); be female (35%); and be 25 or over (61%).
The relatively low proportions of non-American and non-English language students in ALN courses may be surprising at first glance. However, this is probably an artifact of the different modal student status of foreign vs. U.S. students at NJIT. Almost all of the foreign students live on or near campus, and attend full time; their visa requires that they take at least 12 credits a semester. Many more of the American students are full time workers and part time students, living at some distance from the campus; the categories of students who are most attracted to the convenience of ALN.
The approach that I use to deliver distance courses is to always have a distance section offered together with a face to face section. I utilize the CMC system to put both sections together as one class. In fact, many of the project teams turn out to be a mix of distance students and face to face students.
STRUCTURING A LARGE ALN COURSES
I teach a number of senior and graduate elective courses that can only
be offered once a year. It is difficult to exclude students who “need
the course to graduate” when the normal course enrollment maximum of 30
is reached. As a result, I often have enrollments of 40 to 60 in
a class that ideally would have a maximum of thirty students.
For a long time I was able to use the CMC system that we designed at NJIT (Virtual Classroom on EIES: Electronic Information Exchange System), which had a number of features that were useful for allowing an instructor to handle classes of this size when there was active participation by all the students. Since EIES was scheduled to be retired in August of 2000 after 25 years of usage, I began using a commercial system called WebBoard the past year.
The objective of this discussion is to explain how I can use such a system to manage a very large and active discussion group. There are at least six key aspects of successfully leading large ALN courses. Many of these are predicated on the observation that many upper division and graduate courses have a high degree of pragmatic content, which is particularly true of the courses I teach in Interface Design and Management of Information Systems. These six principles are:
1. Information organization and retrieval:
One must carefully structure the activities in a course into different conferences or discussion forums, so that all of the information in one conference pertains to a small number of related topics, and no one conference gets so large that nobody can find anything in it.
2. Synchronization of the class as a whole
Set clear guidelines for what is to be done where and when, and strictly enforce them, to maintain the order that was laid out in the organization of the conferences.
3. Coordination, Collaboration, and Socializing among the members
Motivate, encourage, and facilitate truly active and collaborative interaction among the students. It is also to build up trust and openness in the expression of views. This requires an active and ideally daily presence by the instructor.
4. Sharing of knowledge
Given the usual mix of students with considerable working experience and those without, it is extremely important to have the experienced students try to understand the concepts of the course in terms of their real life experiences and to bring those understandings to the rest of the class. Students then take the pronouncements of the professor far more seriously.
5. Sharing of Learning and feedback
Given the nature of many abstract concepts, the instructor can better perceive if he or she is getting the message across when the students feed back those concepts in their own frames of reference. Also, those representations may be more relevant for understanding by other students than the ones the instructor was using to introduce the conference. This is a form of the Montessori effect.
6. Require participation
Grade students on the quality and timeliness of their contributions (not just for quantity, or going through the motions).
The foundation for being able to handle large ALN classes is certain features in the design of the group communications technology and the interface to the system. I will compare WebBoard features with the features used in the original EIES and point out the improvements needed in most current commercial systems (including WebBoard) to make the task easier and possibly allow extending the size to about a hundred students.
A. A Specific Example
In the spring of 2000 I offered my graduate course in the Management of Information Systems (CIS 679). This is an elective taken by graduate students in the Masters program in IS and in some other areas like Computer Science and Management. It is also required for the Ph.D. students in IS. About half the course focuses on the task of managing software development projects for applications in an organization.
The WebBoard approach is to set up a separate “board” (based upon the bulletin board metaphor) for the course. Within a board the instructor as a manager may set up any number of conferences. Each conference may be of a specific type having certain choices of functions, making its structure different from other conferences. In the old EIES system one could use multiple conferences, but one also could set up at any time in a conference an “activity” which had a unique structure; one of these was the question-answer activity which established a distinct sub-discussion which no one could join until they had given their individual answer. What was common for all these activities, and crucial, was that in EIES, one could get a list of all activities in a conference showing the status of each, including whether the given student had done that activity. While not as complete, one can get in WebBoard a list of all the conferences in the board showing how many items were entered and how many were new.
In WebBoard the manager of the board (i.e. the instructor) can reorder the list of the conferences so those that are no longer active can be put last and those that are new and important to participate in can be put at the head of the list. This provides a useful synchronization cue.
In my use of WebBoard I started the class with the following conferences:
Questions on Assignments
Questions on Book Chapters
Questions on Lectures
Questions on Readings (i.e. professional papers assigned)
Management Jokes (related to IS)
Cafe and Practice
Instructor’s Instructions used the feature that allows the manager to
set up a conference which only designated people can write in. In
this case it was the manager or the instructor, but one can have multiple
managers or instructors. Clearly this conference is for anything
the instructor wants to make sure will not get lost and the students know
they need to have carefully read everything in this conference.
This included the links to the material I have on the web for the syllabus and other notes that are in the video tapes or CD versions of the lectures. It includes descriptions of all the assignments and, of course, an explanation of each conference in the initial list and the objectives of that conference. As we will see, the end of the course shows a much larger list of conferences that have been reordered a number of times.
Perhaps the most important instruction I give them is a need to be sure they put things in the right conference and in the right place in a conference. I inform them I will delete anything that is not in the right place and they will have to reenter it. As an instructor one has to be ruthless about this until the students begin to realize the importance of organization to understanding what is going on. The first few times I delete for a student I actually make a copy in a message and send the contents back to the student as well as deleting it. Rarely do they repeat the problem and most students do make use of the practice conference to understand the system.
The second instruction is to be sure to Reply to a root item rather than creating a new one. One of the problems of WebBoard is that you can only create in one frame a single discussion thread. The display of WebBoard has two frames: a list of conferences on the left and the text of what you are looking at on the right. In the left you can click open a conference to the list of root items and see which have developed a discussion thread. If you click on given root item (they call it a message rather than comment) then it displays in the right frame the contents of the root comment and all the replies to any level to that root item. The most common mistake beginning users make is to post their reply as a new root item and at that point all ability to see a given discussion thread in its entirety would disintegrate.
There are some so-called conference systems which only allow you to view one comment at a time, even as part of a discussion thread. Unless a user can see in a single scroll the whole discussion thread, it is impossible to follow a complex discussion (e.g. when you have to do operations to switch from comment to comment as opposed to just an unconscious scroll operation). If users cannot comprehend a complex discussion because of the interface functionality, then it goes without saying that they will NEVER create or have one. Many people using simpler systems probably don’t realize what is missing and this may be part of why they feel the ALN approach will never duplicate the face to face class.
However, WebBoard is far from ideal in that the original EIES allowed one to link together any sequence of items within a conference and thereby allowed one to merge a number of discussion threads as well as collect relevant items from other discussion threads that might be relevant to a different discussion thread. WebBoard could make a tremendous boost in its utility of dealing with complex material by merely allowing one to select multiple items from the left hand outline frame and allowing the opening of multiple threads to select ranges and individual items in this multiplicity. In the long run discussion structures have to allow non linear Hypertext types of topic organization (Turoff, et. al., 1999).
The Introductions conference is where everyone introduces themselves and at least informs the class of their experience in the field, their objective for taking the course and what specific topics they think at this point they are most interested in. After a root item which is the objectives, everyone replies to that root item so that the whole list of introductions can be viewed as one scroll. I put in a background item giving my interest in the topic and an explanation of why I am qualified to teach this topic. This first participatory conference is important in that it helps create a welcoming atmosphere and to establish “swift trust” (Jarvenpaa & Leidner, 1998), so I make every effort to respond to most introductions. The students also know there are assignments where they can work as teams so they realize their introduction is useful for team formation.
For a lot of conferences I point out we need to make every entry after my objective statement a reply to the root so we can get a scroll (and a printout or copy) of the entire conference for use as needed. So even new topics that would normally be conceived of as “root” items must be a reply to the objective comment as the only root item in the conference. This is a good example of how users often have to find a way to work around the mistakes of the designers of software. It is relevant to the course management problem in that it is often a mistake to assume users will do a certain thing in only one way when developing software requirements.
Separating out the questions into different types makes it easier for the students to find if a question has been addressed before asking it. It is much better than dealing with email for individual students. If I get an email I may copy the question into the conference rather than answer the email if I feel it might be asked by another student. WebBoard does have the desirable feature that you can enter comments anonymously so a student who feels their question might be stupid can still enter it. I actively point out they can enter comments anonymously and they should feel free to do so. Sometimes I will take the role of a student and argue with myself (anonymously) to get this started. Many subjects in Management have multiple viewpoints and it often revolves around very detailed considerations that do not appear until these contrasting viewpoints are explored.
I seed the joke conference with a number of items about management, including some Dilbert cartoons. They are told adding to this is not an assignment and not necessary but it can be done anytime they do run across something providing some useful but amusing insights.
The cafe and practice is for anything they want to talk about and general socializing that may have nothing directly to do with the content of the course. It is the online water cooler or coffee break place. It is important to help in getting to know one another and in developing working relationships.
The above set of conferences is stable for one week or until most of my students finally appear. I do put my face to face section of the course together with my online section and they become one course. Some face to face students do not know this is happening until they come to the first day of class and they need to the week to get oriented and up to speed. In the second week the following conferences are added:
Bad Management Examples
The readings for the course include about seven professional papers
which are available over the Web by password in PDF format. Also
on the server for this course are some classic news articles about major
IS management disasters (e.g. Business Week articles). However, if
I have any additions to make during the course I use a separate conference
for attachments contributed by the students or myself. I also provide
about 50 useful websites I have found and invite students to find others
and add them to the collection.
The first real assignment in week two is to either come up with an example of bad IS management from their work experience or find one written up in a new source. I ask them to try and hypothesize what was the real cause of this IS disaster in terms of management actions or policy. My objective is to respond to most of these with comments pointing out possible alternative explanations from the one they have expressed and to engage the whole class in a discussion about some of the more significant ones.
In week three, two more conferences are added. Some of the course materials deal with doing numerical (regression) projections on key factors in the IS field such as the cost and performance of hardware and software. I provide a number of good examples and then ask them to find an example of a data projection they can do into the future such as percentage of US population using the WEB. These and associated discussion go into a separate conference and there is no fixed due date on this assignment.
Nuggets & Turds
The nuggets and turds conference is set up about the same time and it
is a voluntary assignment. Any where in the course (lectures, readings,
book, etc) that students find something they think provides an outstanding
insight or that they feel is just “bullshit,” they can put it in this conference,
explaining why they have that view.
During the course I will have them do as teams 3-5 Harvard Case studies taken from the text book. They can do these as teams and they hand in a written copy. However, after the written report is due I start a general discussion on some critical aspect. This resulted in four additional conferences in Spring 2000:
Canadian Airlines Discussion
Only two of these were formal case studies and the other two were focused
on some questions for the Ph.D. students and not required for the other
In the old EIES system the case study assignments were done using a question and answer activity so that no one could see the discussion answer of another student until they had put in their answer, and then those that had answered could engage in free discussion. In WebBoard I use the approval conference feature, where no entry can be seen by anyone except the manager of the conference until the manager approves it. Therefore I set a due date when all the first answers have to be in and then approve everything and change the conference from approval to a regular one so free discussion can proceed. This is more clumsy than the original EIES version but still more useful than the ordinary conference structure for stimulating independent thought among all the class members.
I also use the approval conference feature for setting up the following conference about a month before the midterm:
Usually in the first week of this assignment, students are required to come up with an exam question that I could use on the midterm. I commit that I will include at least two student-generated questions on the 10-12 question midterm essay exam. At the end of the first week I approve all the questions so they may be viewed by everyone. I may have done some editing to improve them. Then everyone must find an unanswered question and answer it. Then the person who wrote the question must grade it. I will look on and step in when I feel the grading is wrong. So the whole class will have gone through some 40-60 questions and learned the answers to each before the midterm occurs. I still give the exam but it is almost an anti climax.
Exam Question Assignment
At the end of the course the conference list was ordered as follows; the number in parentheses is the total number of comments in each conference.
Development Process Discussion (223)
Final Discussion (90)
Final Projects (73)
General Discussion (186)
Instructors Instructions (12)
Bad Management Examples (115)
Evaluation Process (39)
Questions on Book (16)
Questions on Readings (4)
Questions on Lectures (91)
Nuggets & Turds (199)
Extrapolation Example (48)
Internet Securities (9)
VeriFone Discussion (61)
Virtual Organizations (8)
Canadian Airlines Discussion (59)
Article Reviews (102)
Project Proposals (38)
Exam Question Assignment (134)
Useful Websites (50)
Useful Attachments (7)
Management Jokes (37)
Cafe & Practice (100)
This class ended up with about 40 students completing and lost about
10 along the way from withdrawals. The total of the above was 1,701
comments, about 113 per week and about three per member per week.
This was over 100 comments per week that a participant had to read.
If they only signed on three times a week this was over 30 comments per
sign on. In fact, I suspect most students signed on every day, only
occasionally missing a day in the week.
Teams could have their own private conference established in the board and those are not included here. I would roughly guess I received about five to ten private messages a week from students in the course. About half of those I transferred to the conference under anonymity. The true private messages dealt with things like travel plans and advisement. After the supposed “end” of the course, comments kept being added to some of the conferences, which I consider one sign of successfully building a learning community.
Actually about 300 of the comments were mine and the distribution of comments among the students were more logarithmic:
219, 128, 108, 90, 61, 58, 50 39, 36 etc.
The modal participation was in the 25 to 30 range with like an average of two comments a week. While WebBoard does not supply statistics on the size of the comments most comments and particularly assigned discussion comments are fairly long (about a page) and clearly there is much thought that goes into the discussion entries. It is easy to see that students have a lot of concern for what the other students think about what they contribute.
B. Final Observations on Teaching This Course Online
Monitoring, grading and organizing all this material requires daily
attention. Certain systems features could decrease the workload.
Systems for learning need better statistics that would show things like
last date online, how often a person started a root entry and how often
he or she replied to other root entries. Perhaps the most serious
shortcoming of WebBoard is that even though the board manager has the right
to edit another’s entry, there is no information provided in the header
that indicates an item has been edited and by whom. There is no alert
to the group that an item they may have seen is now changed.
In the old EIES system modified items were always delivered again as new.
As the instructor I probably had to modify somewhere between 100 and 200
items but could not insure everyone would detect the change even though
I usually made a comment in the item as to what changes I had made.
The use of fixed key words would also be a great improvement where the instructor can establish a list of key words that can be used as retrieval keys for individual items. Indicating comments were: questions, answers, opinions, arguments, etc would be very useful and also allow the instructor to better gage what is taking place on a collective bases. The old EIES system allowed free key words for any comment.
How does one obtain a high level of participation? Besides establishing and enforcing a structure that can accommodate it without causing information overload, one also needs to motivate it. Part of this is direct grading: 10% of the course grade was for “required” participation, and 10% was for voluntary participation. But in addition, the students soon realized that participating actively would improve their mastery and grades; for instance, participating in the exam and case study conferences would improve their grade on the examination. Secondly, an active style of “Socratic” dialogue was established and modeled, to encourage not only frequent participation, but also thoughtful, analytic participation. However, there are always a few students new to this type of course who do not see the importance of participation until they realize their grades are not turning out as they had hoped. All of a sudden, after the midterm, they go back and add their comments to all the discussions they previously ignored, even though you explain that this will not count. I now have a policy of turning certain discussions into a read only conference when I feel it the discussion period for that topic should have completed.
The final serious deficiency for the WebBoard software was the inability to download a transcript of everything or of the items generated in some date range so that the student might have a file on their personal computer they can review. They do have the feature for requesting that each new entry be sent individually as a message to the user, which of course completely destroys any ability to follow the organization of the material. Also at the end of the course a complete transcript is extremely useful for improving the material for the course. With the limitation of only being able to see a single discussion thread, an instructor or student wishing to do this at the end would have to perform at least three to four hundred separate operations of opening each discussion thread, or comment, in each conference, individually and then copy and paste it into to his word processing file on his or her PC.
While WebBoard and other conferences systems, (many of which are even less flexible than WebBoard), are suppose to be general purpose for business use as well as learning, these sorts of limitations show a complete lack of understanding of twenty five or more years of research in Computer Mediated Communications and the needs of groups to deal with complex problems, such as learning, as an effective group process (Hiltz & Turoff, 1993; Turoff & Hiltz, 1995, 1999).
To really be able to carry out the learning methodologies that one would like to apply depends on a great many functional features of thee asynchronous CMC system being utilized. Often the instructor, as user, has to resort to clumsy substitutes in functionality and use them in ways the designers never conceived they would be used. Hopefully there will be a future generation of software that better understands the needs of instructors and the benefits of being able to handle large active collaborative classes.
FUTURE SOFTWARE REQUIREMENTS
The key limitation to the ability to handle large classes in the range
of a hundred students is the very limited discourse structure currently
used in all the current generation of commercial systems. The simple
“comment and reply” structure is very limited in that it produces a simple
hierarchical structure and does not allow integrated topics and multiple
The instructor needs the ability to specify his or her conceptual map of the structure of the subject matter and to allow that conceptual map to be the classification schema for the nodes and the relationships in the discussion. Such conceptual maps can be quite complicated but they reflect the mental model that the instructor is trying provide the student as a framework for learning. If he or she can make that framework explicit and use it to guide the discourse process it will greatly heighten the ability to convey that model to the students and get them to utilize it as a guide to their discussions.
The part of the course we have just described devoted to the software development process may be represented at the highest level by the following non linear discourse structure of the main phases in the process (Figure 2). Note that the development process is cyclic in nature with many concurrent feedback loops taking place. If one tries to simplify the discussion by utilizing separate conferences all these interactions become lost and lead to many tangential discussions in the separate conferences, and relationships across conferences become extremely implicit in nature and difficult to grasp.
Furthermore, each of these phases has a lower level discourse structure as illustrated by the internal discourse structure for the User Request Phase which also follows (Figure 3). Discourse structures are modular in nature and besides the content structures we also illustrate two meta discourse structures that can apply to almost any class discussion being largely independent of a specific content. These are the “problem-solution” structure and the “argument” structure (Figures 4 and 5).
One can imagine that an instructor could be provided common meta structures as part of the software but would also be provided the ability to create their own templates that reflect their particular content domain.
Figure 2: SOFTWARE DEVELOPMENT PHASES DISCOURSE STRUCTURE
Figure 3: USER REQUEST PHASE DISCOURSE STRUCTURE
There are two principal meta discourse structure templates that can link to almost anywhere in the general discussion of the software development process. The first is the problem-solution nesting that should reflect the management thinking of such a process: problems generate solutions and solutions generate problems. Also anything else can generate a problem or a solution.
Figure 4: PROBLEM SOLUTION DISCOURSE STRUCTURE
The links in Figure 4 represent the “generation” of a new discourse element by a prior one. The other discourse meta structure is the argument or disagreement resolution discourse structure.
Figure 5: ARGUMENT DISCOURSE META STRUCTURE
Any issue which could include a problem or a solution can lead to alternative
decisions, actions, policies or practices. For any alternative resolution
there are pro or con arguments and an argument could be pro for one resolution
or more alternatives and con for others. Also arguments can be opposite
in nature. Therefore there may be many links represented by the simple
triadic discourse structure in Figure 5.
It should be clear that using these discourse structures as templates to allow the discussion to be self organizing would go a long way to allowing an individual to understand the relationships among a much larger body of communications than is possible with the simple comment-reply structures in most current systems. Among the consequences of allowing an instructor to design such discussion templates would be:
SUMMARY & CONCLUSION
Characteristics of NJIT ALN Students vs. Other Modes, 1997-2000
Is English your native language ?
Are you a
|Condition||Resident student||Commuter student||Distance student||Total N =|
|Condition||Single||Married||Total N =|
Average number of hours per week that you currently work for pay
|Condition||None||1-30||31-40||40+||Total N =|
If employed, will your employer reimburse your tuition
|Condition||Not at all||Partially||Completely||Total N =|
What is your age ?
|Condition||17-20||21-24||25-33||34-99||Total N =|
What is your academic standing?
|Condition||Freshman||Sophomore||Junior||Senior||Graduate||Total N =|
Of what nation are you a citizen?
|Condition||USA||Other Total||Total N =|
What is your ethnic background
|Condition||White||Asian||Black, Hispanic, Other||Total N =|