MED. INFORM. MARCH  (           2003)     VOL.    28,   NO.  1, 1­5




Evaluation methodology for a medical e-education
patient-oriented information system


               OLIVIER CURE*              ´

               Universite´ de Marne la Vallee, Laboratoire ISIS, 5 boulevard
                                                            ´
               Descartes, Champs-sur-Marne, 77454 Marne la Vallee Cedex 2, France              ´



               (Received February 2002)


               Abstract. This paper describes an evaluation methodology designed for the IMSA
               (Interactive Multimedia System for Auto-medication) system. IMSA stands in the
               multidisciplinary field of medical informatics and aims at providing a health care
               Internet tool for the general public. As far as a medical information system is patient-
               oriented, issues in understanding content and of human ­ computer interface usability
               are key factors to achieve success. The team developing IMSA concentrated part of its
               work on designing an evaluation methodology to ensure that the system is user-friendly
               and responds in a way that can be easily understood by the general public. This
               methodology is built on a three-tier architecture with the user, the evaluator and a set
               of medical case descriptions. The testing protocol is based on a user trying to run
               scenarios related to mild clinical signs on the IMSA prototype under the passive
               presence of an evaluator.



               Keywords: Information system; E-learning; Auto-medication; Usability; General public;
               Evaluation protocols.



1. Introduction
   IMSA is a medical e-learning tool that provides reliable, updated and accurate
information related to the field of auto-medication (the act of treating ones-self
with or without drugs). The development of IMSA is the result of a long-term
multidisciplinary collaboration between health care professionals and computer
scientists specializing in presentation of knowledge [1]. Both teams were concerned
with the system's usability.
   Usability of a computer application can be defined as the ability of the system to
allow the user to carry out tasks effectively, efficiently and enjoyably [2]. Recent
examples of e-commerce have shown that denying design principles of usability
can lead to bankruptcy. Web sites that deliver biomedical information are also fa-
cing this fact [3].
   This paper describes the protocols designed for the IMSA evaluation metho-
dology and proposes an analysis of the results, which will lead to a conclusion on
evaluation methodology for medical e-education systems.



2. Evaluation methodology cognition
   The IMSA evaluation methodology is cognitively-based and requires expertise
in medicine and the human ­ computer interface. Both teams collaborated on the


*Author for correspondence; e-mail: ocure@univ-mlv.fr

  Medical Informatics & The Internet in Medicine ISSN 1463-9238 print/ISSN 1464-5238 online # 2003 Taylor and Francis Ltd
                                                http://www.tandf.co.uk/journals
                                              DOI: 10.1080/1463923031000107785

2                                           O. Cure´


theoretical and pragmatic aspects of this evaluation. This work was divided into
four parts: profiling testers, writing scenarios, designing an evaluation form and
defining a questionnaire for the tester.


2.1. Profiling testers
    The purpose of this task is the identification and selection of potential users of
medical e-education system. We defined four panels of users (table 1) based on a com-
bination of the requirements from computer scientists and health care professionals.
    Studies in usability [4 ­ 6] have emphasised that a group of five testers is opti-
mum to obtain valuable results. We defined the panels concerning internet experi-
ence and openness to auto-medication with a small questionnaire (table 2).
    Our panels were put together considering that:

  . an experienced Internet user is someone with at least 6 months experience of
      the World Wide Web and an average of at least 3 h of browsing a week.
  . a user open to auto-medication is someone who happens to prescribe himself
      medication, does not consult a general physician when he or she does not feel
      it is necessary and is involved in seeking information on drugs and medicine.


2.2. Scenarios
    The scenarios are medical case descriptions. Their purpose is to give a well-de-
fined framework for the tests. The test protocol will be the same for all groups: tes-
ters will be handed a set of scenarios and will be asked to play the patient role
considering this medical case description. The collaborative work once again pro-
vided valuable requirements: computer scientists wanted to design navigation dri-
ven medical case descriptions while health care experts were motivated to write
diagnostic issue driven scenarios. Both teams collaborated to design scenarios that
would fit all requirements considering the major constraint of time. Our previous
experience of usability testing emphasised that the whole test process should not
exceed 25 ­ 30 mins. Exceeding this period, computer technology lay people tend
to lose focus, providing insignificant and unreliable results.



          Table 1. User profiles for the four panels of the IMSA evaluation methodology.

                                                                   Skill in web navigation

                                                                 Novice               Expert

Openness to auto-medication                   Open             Panel no. 1         Panel no. 2
                                           Non-open            Panel no. 3         Panel no. 4




                                Table 2. Profiling questionnaire.

Question number   Question

1                 For how long have you been using the World Wide Web?
2                 On a weekly basis, how many hours do you browse the Web?
3                 Do you self-prescribe some drugs yourself?
4                 Do you read books and or magazines related to medicine or auto-medication?
5                 Do you watch television programme or listen to radio shows related to health care?
6                 Do you consult a general physician any time you feel sick?

              Evaluation methodology for a medical information system                3


    Considering this time constraint, we defined six scenarios leading to specific
health issues: (1) proposing a single drug, (2) proposing a list of possible drugs,
(3) the patient does not need any drugs and is only given advice, (4) patient is
not given any drugs because he or she presents contraindications and (5 ­ 6) asking
the patient to consult his physician because his symptom are too serious (two sce-
narios for this issue).
    All six scenarios could have been related to different clinical signs but we
decided to have just one symptom. We chose `cough' because many forms exist
(productive cough, non productive cough) leading to different treatments. The
heterogeneous rating (from 2 to 18 out of 20) of drugs related to `cough' helped
us to take this decision.
    The cognitive approach of the scenario writing was predominant and we were
also concerned with their semantics. All scenarios should be easily understood by
all patients, not requiring the help of an evaluator.


2.3. Evaluator form
    We designed an evaluator form to structure the notes taken during the tests.
The task of evaluators, who stand behind testers, is to observe the global navigation
and overall interaction on each scenario. The evaluator form consists of an infor-
mation page and scenario forms.


2.3.1. Information page This page contains information about the evaluator: first
and last name, about the testers: first and last name and the test: date, starting and
ending hour of the test.
    The information page also contains two blank parts:

  . Observations at the end of the scenario playing stage.
  . Observations completed after a discussion with all the testers.

2.3.2. Scenario form The goal of this form is to quickly visualise if the tester went
through the desired navigation process of the scenario. Each medical case descrip-
tion has its own page which is made up of the same framework. This framework
consists of:

  . starting and ending hour of the scenario interaction.
  . a table with rows, sorted sequentially, containing all the expected interac-
     tions. Each row has a binary reply, whether the tester did interact properly or
     not.
  . a question : Did the tester reach the expected result?
  . a blank space where the evaluator can write comments.

2.3.3. Questionnaire The questionnaire provides valuable data that could not be
obtained from the passive presence of the evaluator. This questionnaire (table 3)
concentrates on the navigation semantic and understanding of the interaction.



3. Running tests
    We designed the testing protocol considering tester psychology and cost-effec-
tiveness. This usability testing approach is portable, because it can be run in any

4                                          O. Cure´

            Table 3. Test questionnaire asked to testers after running the scenarios.

Question number  Question

1                Do you find the symptom to drug navigation clear enough?
2                Do you understand the meaning of the `Imperative' questions?
3                Do you understand the meaning of the `Diagnostic' questions?
4                Do you understand the meaning of the `Treatment' questions?
5                How would you describe the number of questions asked: too important, sufficient,
                 not sufficient, no opinion?
6                How do you judge `forced exit' of the symptom?
7                How do you consider the way you dialogue with the system: interactive enough, too
                 rigid, too flexible?
8                Do you have enough criteria to choose a medication from a list of drugs?
9                Do you think that IMSA proposes interesting services?
10               Considering that such a system is available on the World Wide Web, would you use
                 it ? Would you trust it?




computer classroom with at least five computers. The main characteristics of the
protocol are:

  . one test administrator, usually one of the evaluators, would quickly introduce
      the IMSA project, present the topic and protocol of this test. Then he or she
      would hand out the scenarios. He (she) would also ask the testers to run the
      scenarios in the predefined order and try not to interact with the evaluators
      during the testing process.
  . one tester per computer with one evaluator standing behind him or her,
      completing the evaluator form.
  . no video recording of the tests.
  . no special electronic devices.
  . one panel group per session.

   Most of the time, the evaluator team would be composed of two computer
scientists and three experts in medicine. The evaluator team would run the ques-
tionnaire immediately after the test stage.



4. Results
   The results from the evaluator forms and questionnaires are valuable sources of
formal and informal data.
   The formal data from the evaluator forms and are summarised in table 4. The
results are emphasized that all scenarios were run successfully. The fact that ex-
perienced and novice Internet users were all successful on the scenario navigation
proves the adaptability of the web architecture.
   The informal data emerged from the comments on the evaluator forms and
questionnaires. This information is valuable to understand some of the results of
table 4.



5. Conclusion
   Designing and conducting this evaluation methodology was valuable for the
IMSA project. The analysis of the data helped to redesign some aspects of the sys-

                Evaluation methodology for a medical information system                             5

                       Table 4. Results from the analysis of test questionnaire.

Question    Results from all 20 users Analysis of results
number

1           100% affirmative
2           40% affirmative                              `imperative' wording is not explicit enough
3           90% affirmative
4           100% affirmative
5           90% sufficient, 10% not sufficient, 0%
            important, 0% no opinion
6           65% of users wanted more information.        All users from panels 1 and 2 wanted more
            35% of users were satisfied on that point    explanations when asked to consult a general
                                                         physician.
7           85% interactive enough, 15% too rigid,       The `too rigid' replies are users from panels
            0% too flexible.                             2 (1 user) and 4 (2 users).
8           70% affirmative, 30% not affirmative
9           100% affirmative
10          On using such systems: 65% affirmative,      On using such systems, replies not
            35% not affirmative                          affirmative are from panels 3 and 4.
            On trusting the system: 80% affirmative,     On trusting such systems, replies not
            20% not affirmative                          affirmative are from panel 4.




tem (`imperative' wording, explanations on drug rating and drug categorization,
explanations on some system's decisions, etc.). Apart from implementation consid-
erations, this evaluation is also a great opportunity to communicate with potential
users on the domain of study.
    Our evaluation methodology demonstrates that you can run a usability study
without great financial investment. The most important aspect of any usability
study is to test with potential users. We think that using video recordings and high
technology devices would not contribute to more reliable results.
    This evaluation methodology framework is portable and could be used in pro-
jects not necessarily related to medicine, but where cognition is a key factor. The
three-tier architecture can be used in any discipline and requires expertise in us-
ability and the specific domain of study. A cognitive approach is necessary during
the designs of the panels of users, questionnaires, forms and scenarios.
    On the medical e-education issue, it would be of interest to study patients ex-
posed to such computerised tools and specially study, over a long period of time,
their relationship with doctors.



References
1. CURE´ , O., LEVACHER, M. and GIROUD, J. P., 2001, Interdisciplinary collaboration for a patient
   oriented medical information system. Proceedings of Multimedia and Technology Application Con-
   ference, IEEE MTAC `2001, University of California, Irvine, California, USA. November 2001,
   pp. 70 ­ 76.
2. PREECE, J., ROGER, Y., SHARP, H., BENYON, D., HOLLAND, S. and CAREY, T., 1994, Human ­
   computer interaction (Boston: Addison-Wesley publishing company).
3. SACILE, R., WILEY, T. and LOMBARDO, C., 1999, Quality assurance guidelines for a biomedical in-
   formation web system: the working experience of the BreakIT project. Medical Informatics and the
   Internet in Medicine, 2, 109 ­ 120, 24.
4. NIELSEN, J. and MACK, R. L., 1994, Usability inspection methods (New York: John Wiley & Sons).
5. RUBIN, J., 1994, Handbook of usability testing (New York: John Wiley & Sons).
6. NIELSEN, J., 1993, Usability Engineering (New York: Academic press).


