A Decade of Research: Assessing Change in the Technical Communication Classroom Using Online Portfolios

Journal of Technical Writing and Communication, 2006, (36:4), pp. 413-431.

Over a period of ten years, we have developed a sustainable process of online portfolio assessment that demonstrates both reliability and validity, using both qualitative and quantitative measures. The sustainable cycle is that, each semester, we assess a random sampling of the students’ work that they have posted, as per our instructions, in an online portfolio. During the reading, faculty score the documents for twelve variables, including writing, content, audience awareness and document design. We achieved validity by a modified online Delphi that led to a redefinition of the construct of technical communication itself and we achieved reliability by adjudication resulting in adjacent scores. The results of our assessment meet the requirements of ABET and result in continual cycle of improvement for our technical communication curriculum. It has also resulted in an improving correlation between the course grade and the separate overall portfolio score – we are beginning to grade the variables that we have defined. For ordering, see: Baywood Publishing.

The Analytic Assessment of Online Portfolios in Undergraduate Technical Communication: A Model

Journal of Engineering Education, October 2006, (95:4), pp. 279-287.

This paper describes an innovative model for assessing the technical communication course by analytically scoring online portfolios, open to the internet, for ten separate (analytic) variables and one overall (holistic) score. The model is a statistically verifiable and sustainable method that strengthens the curriculum and fosters consensus within the teaching community. We achieved construct validity by redefining the elements of the course to incorporate communication in the digital age and then by creating new criteria for evaluation related to that construct. We achieved inter-reader reliability by beginning each assessment with a calibrated reading and by adjudicating non-adjacent scores. After using the model successfully for three semesters, we can see increased consistency in teaching among sections and semesters, more communication among instructors and we are beginning a database with which we can test further change. The theory and method behind this model can be applied to other disciplines as well. For reprints, email cjohnson@njit.edu.

For more information see:

• Online Portfolio Assessment Website
• College Writing Assessment