This report is about Virtual Classroom delivery of CIS - 114, the introductory Algorithms and Data Structures class. First I will describe the class itself, and then go into the particulars of my remote class sections.

CIS 114 at NJIT corresponds to the ACM curriculum CS 2 class, i.e., it is the second Computer Science class for majors. Anybody who is enrolled in CIS 114 is expected to have taken CIS 113, or an equivalent course, as prerequisite. CIS 113 introduces the students to the basics of PASCAL programming, leading up to a (superficial) coverage of linked lists. Students of CIS 113 learn design, editing, and debugging of programs by hands-on experience.

In CIS 114 this elementary knowledge of Computer Science is then extended in the following ways: (1) The notion of Abstract Data Type is taught. (2) Standard Searching and Sorting Algorithms, such as Quicksort, Mergesort, Radix Sort, etc. and the commonly used mathematical formalism to describe the order of magnitude of their runtimes (the big Oh notation) are introduced. (3) Linked Lists, Stacks, Queues, Trees, and Graphs are covered as Abstract Data Types. (4) Some of the more intricate details of PASCAL are covered, such as use and misuse of variant records. (5) Topics that are traditionally not well understood by students of CIS 113 are reviewed, especially the use of recursion.

In the Spring of 1993 I recorded 40 half hour lectures of CIS 114 on video tape, using the staff and facilities of the Candid Classroom at NJIT. The normal scheduling practices at NJIT resulted in one two hour session per week, during which three half hour segments were recorded, and a second somewhat shorter session every week, where some topics were explained and extended without the ``threatening'' camera above everybodies heads.

During the Fall of 1993, I taught a first CIS 114 Virtual Classroom course. In this course, students would see the lectures on video tape or cable, and they interacted with me by using the EIES conferencing system. In the Spring of 1994 I taught the same class again, and an additional class called a ``double speed section.'' This section started in the middle of the semester and continued for (a little over) half of the normal semester.

In Fall 1993, CIS 114-451 had 8 registrants. Unfortunately, 5 withdrew, and one failed, leaving only two passing students (with grades A and B).

In Spring 1994, CIS 114-452 had 13 registrants. Of these three dropped the class. Of the remaining students two received A's in the normal time. Eight students received incompletes. Of these eight, seven finished the course within the allocated time to make up snow and tape problems. They received the grades: A, B+, B (2) C+, D, F. One student is currently making up this class.

In Spring 1994, CIS 114-454 had nine registrants. This is the double speed section. Of these, four registrants dropped the class. Three remaining students made use of the "short term incomplete" option and received the grades A and B (2). Two students have to date not made up their incompletes and are expected to do so in the Fall semester.

For purposes of comparison I will list now the data on a CIS 114 delivered in the normal face-to-face mode, with no EIES and no Video support. This section was given in Fall 1993, at the same time as CIS 114-451 mentioned above. It's number is CIS 114-001. This class was registered by 33 students. Of these 33 students, 19 dropped the class, which is unprecedentedly high, even for a difficult class such as 114. In my estimate, a "normal" drop out rate for 114 is about 40%. To the surviving students, the following grades were given: A (5), B+, B (2), C (3), D, F (2).

In the balance of this report I will discuss experiences made during those three sections, concentrating on the Spring 1994 ``normal speed'' class, but making comparisons whenever it seems appropriate.

The basic format of CIS 114 in the Virtual Classroom was as follows. Students were required to sign onto their computer at least twice a week, preferably after seeing the appropriate three video tapes of the week, and after reading the corresponding book chapter. (Of course, there was no good way for me to verify this.) I would post three questions every week, on the material of the chapter, and students were required to supply answers within four days. I repeatedly pointed out to them, that the three questions would NOT cover all the material of the chapter. Rather, these questions stressed important topics, pointed out the expected depth of study, and prepared the students for possible question formats at their exams.

Some of the questions were posted as simple conference comments, but most of them were posted as ``activities.'' An activity is a posting for which a student has to post a reply before he can see the replies of the other students in the class. Students were also asked to comment electronically on at least one posting of another student within the two days after the first weekly due date. This posting could be a criticism, a correction, or even a question for clarification.

In addition to these requirements (``interaction homeworks'') students also had to mail in printouts of three programming assignments. The assignments themselves were posted on the EIES system. Midterm and final exam were given in class. Even though the Distance Learning Center and the Project Director permit the use of proctors for exams, I consider this detrimental to the reputation of the Virtual Classroom, and I insisted on being personally present during the exams. As none of the students was from outside of the three state area (New York, New Jersey, Pennsylvania), this requirement did not place an undue burden on the students. In fact, most of the students appeared to be within about an hour driving distance from NJIT.

During the Spring of 1993, two situations occurred, over which nobody at NJIT had any control, and which caused some problems. The first problem was an uncommonly cold winter, with an immense amount of precipitation. This lead to several snow closings of the University and, interestingly, to the fact that the Virtual Classroom courses were almost entirely on schedule, while most other classes were limping behind. Still, individual students were hampered in their class contributions.

The second problem was with the company in charge of tape delivery, which did not perform very well. Several students had not received their tapes well into the semester, and the Distance Learning Center was forced to switch supplier in the midst of the semester. Some students were very vocal about this problem, one of them going all the way up to the president. In response to this complaint, the administration of NJIT extended the semester for all affected students by about two weeks.

While this decision was in the best interest for the students, it created a number of administrative problems. It did not seem fair to extend the semester only for the ``complainers,'' and it was also difficult to retroactively verify who had had problems with tape delivery (and/or snow!). Therefore, I was forced to offer the semester extension to all students. All but two students of the ``normal speed'' section, and all students of the double speed section took advantage of this extension. Due to this fact, the double speed section was not really a double speed section, and students had more time than they themselves expected for the last programming assignment and final preparation.

1) Problems

(a) Problems with Regular Login
Instructors were prepared by project group discussions and by their own experiences as students and teachers, that both, the most crucial and most difficult problem would be to enforce the weekly login schedule of all students. In practice it was found quickly that only a minority of the students in all three sections kept completely to the schedule. However, the problems and reasons for not doing so were surprisingly varied.

* Hardware problems. One student did not sign on during the first two weeks of the class, so I phoned him at home. He described to me that his computer was right now in many little pieces on his desk. He could not afford to have it professionally repaired, and the chips that he needed for a self repair had not been available to him for some time. He eventually managed to sign on and catch up with most of the class.

* Erratic schedules due to jobs. One student had a software release emergency in his job. Due to that fact he asked for extensions of the online assignments. I should note that I was very generous in such extensions, far more generous than in any face-to-face class I had ever taught. The student was able to catch up ``mostly,'' but some of the answers were obviously done in a hurry, and they did not deserve full (or not even partial) credit.

* Erratic schedules due to family ``problems''. One of the students was a Medical Doctor, who took the course for his personal enrichment. He did very well during the first several weeks. (As a side note, it would have made me rather uncomfortable had I received any ``stupid'' answers from this student, but luckily this was not the case. I guess we can trust in the MD education.) However, before the middle of the semester, a daughter was born to the doctor, and he withdrew from the class after trying to keep up for a little while. Obviously, he was not in a do or die situation, he has a degree and a job, so he could afford to drop the class.

* Initial discouragement due to software problems. Some students seemed to have initial problems with the system. Several students reported to me that they were logged out by the system in the middle of their sessions. In some cases, this seems to have created discouragement and delay. Although I can be trusted to have used the system more often than all my students during these two semesters, it happened to me only once or twice that I was disconnected by the system. However, the reports of the students cannot be disqualified as exaggerations, as my computer/modem is in the same local calling area as NJIT, while many of the students had to make toll calls.

* Just not accepting the facts. Unfortunately, there was at least one student who either ``just did not get it'' or did not want to understand that he was required to do work on a weekly basis. Writing a student a note that he has to read the questions on the system does not help a lot if he does not read/remember/accept the note in the first place.

(b) Problems with Filing Away Information
It became very clear during this course, that students had difficulties with filing away information for later use. ``File away'' can be done either mentally, or physically, by saving information in a private conference, and referring back to it. This is a problem that also occurs in face-to-face classes. Nevertheless, for the teacher it seems more acceptable if a student just forgets something that he heard (or possibly did not hear) than if a student does not remember something that he (claims he) read and that still exists in writing on the system.

This problem occurred specifically for late penalties. Those were posted on the system, but students still came back and asked questions about them. My generalization of this and similar experiences is that online information should be given in small portions and as near as possible to the time of use. Fairness dictates that late penalties be announced at the beginning of the semester or with a project assignment, but (for some students) they better be repeated on the day before the due date. Similarly, announcing an exam date four weeks or longer before the exam is necessary to let students schedule rides, baby sitters, or whatever they need. Still, three days before the exam a reminder is essential.

Roxanne Hiltz, the project director of the Virtual Classroom, has suggested that a possibly way to improve the ability of the students to file away information on the system would be to spend more time during the face-to-face orientation, explaining how to use features such as keyword search.

(c) Problems with getting a discussion going
In project group discussions, the importance of starting interactive discussions with the students and between the students was repeatedly stressed. By specifically requiring a comment on another student's posting, and by grading this comment, I tried to achieve this ideal, but by and large I must admit failure. Some students posted comments some times. Many students never posted any comments, and nobody got full credit on this component of the course. My repeated reminders and requests did not help.

After thinking about the problems, there seem to be four sources for this difficulty:

(1) Algorithms and complexities of algorithms are not up to discussion. You know them, or you don't. There is very little to discuss. Most of the questions that could be raised in a discussion would be either research issues at the "upper end" or the discussion is a result of insufficient study of the material at the "lower end". In the latter case, the students are well aware of the problem and therefore not comfortable with being involved in a discussion that shows they don't know some material.

(2) The pressure in this course is considerable. It takes most people a long time to understand the intricate data movements of Quicksort or Insertion Sort. The programming assignments are also difficult. Students make the reasonable decision that they better answer the required questions and debug their programs before they afford the luxury of a discussion.

(3) Possible there is a third problem that is due to the teacher. I was educated in Austria until age 24. Austria has what I would call a German system of education: The professor lectures and the students listen and take notes. Few students dare to ask questions in a lecture hall with 200-1,500 (!) students. Most of them are not up to the point of asking questions, and professors by and large do not encourage the asking of questions, especially in these large sections. (There are some smaller recitations and seminars also.)

(4) Students seem afraid to critique a classmate's postings, because they are afraid they will cause a lower grade for a fellow student.

2) Lessons Learned

* The Conferencing System is Essential The use of "activities" that permit communication but make cheating impossible is an essential tool in running the Virtual Classroom. In fact, I was approached by students who were worried that other students would just copy their own postings without investing the same amount of time and effort. The use of activities was very comforting for them.

* The good do well. It appears that students that are doing well in general are doing well in the Virtual Classroom, too. Unfortunately, it is hard to supply enough observations to support this lesson, as my personal interaction with most students was extremely limited. It might be interesting to collect complete grade profiles for all students in the VC program, to substantiate this observation.

* Self-discipline is a good indicator for success. The Virtual Classroom appears to be ideal for disciplined students. Again, this is a personal impression that is hard to quantify or even justify. Maybe we could collect data by interviewing previous teachers of students attending the Virtual Classroom. To avoid biases, the interviewing should be done by people not instructing the respective students.

* Face-to-face orientation on EIES boosts survival rates. The most important improvement between the Fall 1993 class and the Spring 1994 classes was the scheduling of a face-to-face orientation in a computer lab, where students were shown the actual EIES system. I attribute much of the improved results between those two semesters to this orientation session. This session took only two hours and was NOT hands-on, except for the instructor, but by projecting the EIES screens onto the wall the ideas came across very well. * New ideas on getting a discussion going. During a project group meeting near the end of the Spring 94 semester, I picked up an improved approach that could help me getting an online discussion going: When a students posts a wrong answer, ask another student (by name) whether he can correct the answer, and give a hint what the problem is. I had previously avoided to "call on a student by name" because I wanted to avoid potential embarrassment. However, in an asynchronous medium such as EIES, the student has sufficient time to prepare his answer and avoid an embarrassment. I had carried over a habit from face-to-face classes to virtual classes which appears overly cautious.

* More ideas on getting a discussion going, suggested by the Project Director. Students could be required to post at least one message every week, that is not related to a homework assignment. It might be related to the course, but it might be more general. For instance, students might have read something in a news paper that appears relevant to the topics of the class.

* Scheduling of questions can be optimized. Students asked me to enter questions and set due dates so that they have at least one weekend in between. This is a very reasonable request (at least for the ``normal speed'' class) but I missed doing this.

* Scheduling of broadcasts can be optimized Students, even remote students, want to take off for Spring Break. My initial schedule did not allow for that, because the cable TV broadcast schedule did not allow for it either. I have already requested from Distance Learning to schedule a "rerun" for the week of Spring Break.

3) Assorted Observations

* The original intention of the double speed section was to cover exactly as much material as in the normal speed section. In reality, the pressure was too big for the students and the instructor, and I settled for dropping one programming assignment and the midterm.

In the middle of the double speed class, students were asked whether they would prefer a midterm and a non-comprehensive final, or no midterm and a comprehensive final instead. The students voted against the midterm. One student pointed out that the programming assignment interfered with study time for the midterm. The bottom line is that programming is such an intensive activity that one cannot assume that it can be done in double speed. Numerous ``all-nighters'' that almost all successful CS students seem to have gone through, support this claim.

* It is sad that during the previously mentioned two hour orientation the car of one of the students was stolen. (It was recovered with very minor damage a few days later). As somebody drily remarked, ``Now I see another advantage of distance learning: I don't have to come to Newark by car.''

* Many students sit through traditional lectures and don't seem to understand many of the ideas. Still, they apparently pick up enough terminology and ideas that they can prepare from their notes and books for a passing midterm grade. In distance learning as we are doing it, students are forced to understand difficult ideas every week, or they won't be able to answer (my) weekly questions. This investment pays off at exam time, where they already know the material and don't have to study much more. It would be an interesting empirical question whether those students will be better in the long run, because they spread the learning over a longer period of time, limiting cognitive interference.

* As a teacher I feel that if I am lecturing the same material several times, I am getting better and better at it. As I am not lecturing during distance learning, I am concerned whether this improvement effect still exists. This question might also be interesting for empirical research, although difficult to answer due to small sample sizes.

* In traditional classes, the number of students that are attending a class is almost irrelevant, except for the time it takes to grade exams. (My programs are graded by TA's.) In the Virtual Classroom, every student adds a load, and I would keep class sizes at maximally 12, if that were economically feasible.

Without claiming any scientific support for the following statements, it appears to me that the Virtual Classroom is an ideal learning tool for students who are highly motivated and/or have a large amount of self-discipline, AND additionally have some previous knowledge of EIES or another conferencing system. My recommendation for a distance learning curriculum of study is therefore to teach the first year, or at least the first semester, in a face-to-face mode. During this time students would get fluent in using EIES and a primary programming language such as PASCAL or C++. Then they will be successful in the remaining completely virtual course of studies.