Cycles of Improvement: Assessing Validity in Technical Writing Programs Using Online Portfolios

Hosted by Rose Hulman Institute of Technology and ABET.

New Jersey Institute of Technology has been involved with writing assessment since 1977, when we were a collaborative creator of the College Basic Skills Placement Test (NJCBSPT). Since then, we helped to develop the first ABET accreditation and have published about assessing technical writing in the Journal of Technical Writing and Communication and Technical Communication Quarterly (Coppola, Elliot). Presently, we are engaged in Program Assessment for continuous improvement. In the field of Technical Communication, we are assessing online portfolios.

Our process of assessment for this course evolved from a best-paper reading, providing longitudinal statistics to chart program change, to an assessment of both visual and written communication in online portfolios. Our methods are both qualitative and quantitative. After the semester, the technical communication instructors meet, read and discuss six pre-selected sample portfolios. This qualitative calibration ends with consensus that leads to high inter-reader reliability. This conversation is recorded and an analysis of its components reveals new criteria for assessment, since, as Bob Broad writes, “The age of the rubric has passed” (4). These criteria are mapped to a matrix that provides detailed descriptors for what we value (and thus what we should teach). This emerging matrix is what leads our development of the course.

After the recorded qualitative session, a randomized sample of the student’s online portfolios is scored with a rubric based on a six-point scale. Each portfolio is read and scored by two readers, neither of whom is the student’s instructor. These statistics provide the basis for charting the curriculum change as evidenced in the student’s work.

This loop of continuous assessment, criteria modification and curriculum change has resulted in a series of assessment documents. These are the tools that we use in this process: a manual for teaching, “English 352: Technical Communication” (that can act as a model for other programs to follow); the criteria matrix (rubric) that names the basic elements of technical communication; our sampling plan, based on a confidence interval; and a reporting structure that includes a histogram and provides the bottom-line assessment.

NJIT has been deeply involved in the process of assessment from the 1980s to the present. For us, assessment is not a gatekeeper but a tool for collaborative faculty activity, program growth, evaluation and change. Assessment has helped us navigate the change from a paper-based writing-intensive program to one that incorporates the news skills of graphics, multimedia, and information architecture that are necessary skills in this information age. Ultimately, assessment is a way of finding and meeting the needs of our students.