S. Schulz, R. Klar, T. Auhuber, U. Schrader
(Abteilung Medizinische Informatik, Universität Freiburg),
A. Koop (Institut für Medizinische Statistik,
Informatik und Epidemiologie, Universität Köln),
R. Kreutz (Institut für Medizinische Informatik,
R. Oppermann, H. Simm (GMD-FIT, St. Augustin)
This catalogue of quality criteria for electronic
publications in medicine was elaborated by the working group "CBT" of the
German Society of Medical Informatics, Biometry and Epidemiology (GMDS).
It is the result of the authors' long-term experience with electronic media.
The motivation for this catalogue was our perception
that many of these products have not yet been developed to a satisfactory
level of quality. Our call for better quality relies on the assumption
that the chances for the success of electronic publications lie in the
added value of motivation and efficiency, dealing with medical information
and acquiring medical skills and knowledge.
Our criteria should not be interpreted as "knock-out
criteria". Representing, preferably, an end users' (or clients') point
of view, our remarks are intended as guidelines for software development,
as well as a basis for the elaboration of evaluation catalogues. We emphasize
that the set of criteria to be used is highly project or product specific.
In the following text the common formulation of quality-criteria
is specific to medical applications. Please note that this does not influence
the applicability of quality criteria to other areas. Many items can be
sensibly applied only to didactic applications, others only to multimedia
programs and so on. This can normally be understood from the context in
which an item is found.
Many items arise from concrete negative examples.
This has had an influence on the degree of detail of the catalogue. Although
some criteria may seem trivial they were included because they are frequently
disregarded resulting in serious, negative and debilitating consequences
for the user. In contrast, some aspects are handled more generally as they
are not specific to electronic publications in medicine and as there exist
other guidelines and quality criteria (e. g. media production, validity
of medical contents).
This catalogue makes no statements as to the value
of the use of electronic publications in medical or nursing curricula,
in patient education or in the Health professional workplace. If a curricular
use of electronic publications is considered, or if the development of
educational software is planned, the relevance and the usefulness of such
a project must be analyzed and an implementation concept should be set
up even before the consideration of quality criteria.
Similar considerations are necessary if the use
of electronic publications in medical working environments is intended.
This catalogue does not aspire to completeness. As
new technologies arise and influence how education strategies (CBT etc.)
can be implemented this catalogue must be updated. For this purpose feel
free to contact email@example.com
for hints and bibliographical references.
The following introductory chapter defines some terms
before the catalogue proper starts in chapter two.
|The development of good EPM requires:
The following criteria catalogue is divided in the subheadings
contents, technical aspects, coding of Information and Modalities of Presentation,
Ergonomics and Design, and Dialog and Didactics. These items may overlap.
competence in software engineering,
Quality criteria that apply to the contents of EPMs
are mainly the same as for conventional publications.
Contents are presented correctly and comprehensively.
The content is adapted to the specified target group
Personal opinions are explicitly marked.
Authors, editors, release dates and version numbers
Quotations and reference sources are marked as such,
attributed and documented.
Medical contents are linked to existing coding schemes
and professional nomenclatures.
The contents are kept current through periodical updates.
Knowledge-based systems are periodically updated by domain experts.
Commercial publications are either internally or externally
reviewed and evaluated as in normal professional, scientific practice.
2.1.2. Formal Requirements
Commercial publications are listed in catalogues and
can be retrieved by their ISBN number.
Copyright information is available.
License agreements contain explicit information and
permission with regard to multi-user operation, loan and rental or purchase.
2.1.3. Target Group Reference
Target users, prerequisite skills and learning objectives
are adequately specified.
If any computer knowledge is necessary, the kind of
knowledge and its amount is clearly specified.
Scope and profundity of the contents are perceptible.
Where standardized curricula exist, it is referenced
in educational software.
2.2. Technical Aspects
The application is developed for the computer systems
available among target users. Ideally, the more common systems are supported,
to ensure as widespread use of the application as possible.
The system requirements and software constraints are
Applications only use fixed graphics settings (i. e.
fixed resolution or color depth) if unavoidable.
Simultaneous usage on multi-user operating systems is
2.2.2 Hardware Constraints
Hardware better than the minimum required (monitor,
graphics adapter) does not impair the quality of presentation.
Hardware poorer than expected causes a warning to be
2.2.3. Software Installation
The application can be started directly from the data
medium without a setup routine.
The application runs without modifying system areas
of the operating system.
There is no need of system reboot or manual modification
of the configuration to start the application the first time.
Where an installation routine cannot be avoided, all
system modifications are clearly documented and an uninstall routine is
The software can be installed on a file server without
the need of separate client installations.
The software runs as a real client/server application
with clients from different platforms (this applies mainly to databases).
2.2.4. Performance / Flexibility / Runtime Characteristics
The application is stable, robust, reliable and performs
In the case of applications designed for quick information
loading delay is minimized.
Where considerable response delays cannot be avoided,
warnings are displayed.
Saving of user-specific settings, data and extensions
(e. g. annotations) is supported by distinct and protected user profiles.
An auto-save function saves user-specific data recorded
during runtime, in short intervals.
Interfaces with complex systems, such as hospital information
systems or text retrieval systems are well defined and sufficiently documented.
2.2.5. Special Criteria for Internet-based Electronic
There is a sensible compromise between ease of operation,
user-friendly design and runtime performance. Realistic data transfer rates
are taken into account.
Large pictures can be previewed ("thumbnails").
Security issues are taken seriously when plug-ins are
requested. Plug-ins are not used where there exists a good alternative.
Internet-based publications that do not communicate
with processes running on the server have the option of an en-bloc download
of the entire package for off-line use.
Intelligent combinations of on-line and off-line elements
aimed at a reduction of communication costs are supported, as much as possible,
when allowed by the content.
Where "gateways" to other Internet publications are
intended the user is informed how to return to the original document.
A "go back " or "return" control is always available.
Those pages that belong to a defined Internet-based
publication contrast by their appearance with other pages in order to minimize
the risk of getting lost.
New browser windows are opened only if there is a real
need and does not occur in an imperceptible way. It is best to maintain
a distinct style difference between the main and all secondary windows.
Due to the volatility of the content of Internet-based
publications and the relationships between internally and externally referenced
WWW documents, the date of the latest update is contained on every page
and modifications are documented. The consistency of external links is
warranted by the author.
2.3. Coding of Information and Modalities of Presentation
Generally, text is more readable on paper than on a
screen. Therefore the computer-based publication of large amounts of text
without additional functionality must be justified by availability, up-to-date
information, retrieval needs and costs, or other requirements that are
not met if published exclusively in a paper-based form.
220.127.116.11. Meta Information
The total size of the publication is transparent.
For each text unit (chapter) the total number of pages
A hierarchical order of the elements in the document
is supported by a logical numbering system.
If there exists more ordering principles, these are
clearly displayed and can be easily accessed. One of them is marked as
Meta-information, such as authors, version number, summary
can be viewed at any time without the need of leaving the current page.
18.104.22.168. Formal Aspects
The contents are expressed concisely and compactly.
Texts are orthographically, grammatically correct and
stylistically consistent, and punctuation is correct.
The elements of composed documents can be selectively
addressed (e. g. for printing, saving and mailing).
Layout, font choice and formatting show consistency.
Larger amounts of text use easily readable fonts, whereas
small, highlighted text units may exhibit more a creative, individual typography.
Tutorial systems do not appear with screens full of
text. Scrolling is avoided, as far as possible, and the rule "one topic
- one window" is followed whenever possible.
22.214.171.124. Acronyms, special terms
The use of acronyms is limited to those commonly understood
in the subject or specialty area.
Where the training or retrieval of foreign terms is
given special emphasis, an acoustic support of the pronunciation is available,
or at least a phonetic spelling using the international phonetic alphabet
(IPA) is given.
Rarely used terms are explained by a glossary or a lexicon.
A glossary can be extended by the user.
A hypertext based publication has powerful orientation
tools that can be used to intuitively and intelligently navigate though
The "hyperspace" of the publication can be graphically
visualized. Direct access to the information about the orientation tools
Hyperlinks are used carefully and parsimoniously. The
semantics of hyperlinks is explained (typed links).
Where sequential reading can not be assumed, implicit
textual references to previous discourse objects are avoided or made explicit
by hyperlinks (as in frequent use of pronouns, or phrases like: "see above"
Advanced organizers (index pages with many hyperlinks)
contrast stylistically with text pages (few or no hyperlinks).
Presentations make distinctly clear whether they offer
loosely connected elements (menus, indexes) or whether they must be received
as a whole, self-contained document.
The layout of "visited links" contrasts (normally by
color) with that of other, not yet visited links.
The individual navigation paths are recorded in a "history"
that can be viewed and stepped forwards or backwards.
Configurable bookmarks and notepad functions are provided.
Users can search the complete content of EPMs with efficient
retrieval tools. These tools take the user's behavior and requirements
into account and, as an option, adapt to the user's interests during run
Retrieval tools support both index based and free text
searches. They can be extended with Boolean operators and wildcards.
Retrieval results are sorted by relevance.
The usage of retrieval tools is explained in an comprehensive
on-line help with examples.
Retrieval methods are able to process synonyms.
Retrieval methods take into account that medical terms
do not always conform to orthographic norms.
2.3.2. Graphics, Animation, Photographs, Video and Audio
The quality of graphic elements, pictures, video and
audio sequences meet professional standards.
126.96.36.199. Graphics and Photographs
The choice of picture/graphic resolution and color depth
is constrained by content on one hand and by the hardware and software
commonly available among target users on the other. In spite of these compromises,
pictures must be able to relay meaningful information.
Zooming pictures offer more visual information than
a simple magnification of the single pixels.
Pictures are labeled with unique, context-independent
188.8.131.52. Animated Pictures and Video Sequences
Animated sequences and video clips are mainly employed
to enhance information acceptance and where they relay information better
than with still pictures or text.
Animated sequences and video clips can be used to produce
humorous or dramatic effects in the context of a thoroughly well planned
user-interface design. They do not produce undesirable delays or interruption
of didactic sequences.
Movie-like opening and closing sequences, when used,
can be skipped over or switched off.
It is best to employ video sequences only where the
user's computer has high enough performance graphics capabilities which
can be relied upon. Where this can not be guaranteed, there is the danger
that the video content may be looked upon as of little use or amateurish
and thus may not be taken seriously.
184.108.40.206. AUDIO Encoding
Where medical acoustic phenomena (auscultation, percussion
etc.) are to be portrayed, the use of audio clips, not only text or graphical
representations, is welcome and important.
Sound tracks, as with video sequences are used only
when they motivate interest or promote concentration without causing undue
Volume and tone can be adjusted or switched on/off.
Spoken sequences can optionally be displayed as written,
on-screen text (such as in subtitling) and can be retrieved. Text that
has already been listened to is marked as such on the display screen.
220.127.116.11. Interaction between Different Modalities of
Contents, target groups and didactic concepts determine
which media are best suited for the project.
When passive sequences require the user's extended attention,
the approximate time needed for the sequence is indicated beforehand.
At the beginning of an application, users are informed
of the media they will encounter.
Extensive, pre-formulated dialogues (doctor - patient
etc.) are not only displayed as written text, but they are also available
as audio documents and supported with pictures or videos, when appropriate.
The division between active elements: browsing, simulation,
didactic dialogue; and passive elements: sound, video, slideshow-like presentations,
reading of plain text, is discernible within the application.
Longer passive sequences are launched actively by the
user and can be interrupted and aborted at anytime. There are several re-entry
points at which the sequence can be restarted.
Switching between active and passive sequences occurs
within a script or concept which is plausible for the user.
Complementary presentation of information (e. g., spoken
text accompanying an animated sequence) is used to increase comprehension.
In didactic applications the use of different forms of presentation is
preferred to merely textual presentation where this supports comprehension.
2.4. Ergonomics and Design
In numerous GUI (graphical user interface) guidelines,
detailed - occasionally even contradictory - specifications can be
found. These will not be itemized in this paper. Instead our recommendations,
which are of special relevance for EPMs (and which are often ignored) will
2.4.1. Basic Requests
For the operation of the system/application assume no
special data processing know-how is required from the user.
The application can be terminated from any place at
The user interface takes into consideration the usual
GUI standards, with which the users are familiar.
The greater the need for interaction, the more important
are GUI conventions.
Basic functionality is self-evident - even without previous
training or the consultation of the online help.
Mouse functions can also be initiated with keyboard
Rarely needed functions are able to be called by using
the standard menu bar or floating palettes, depending on computer platform.
Icons and buttons use plausible metaphors.
The functionality of a control is easily understandable.
Features and design elements of standard software used
by the target group in the everyday life (office suites, web browsers,
email applications, operating system functions etc.), are, as far as possible,
integrated into the design of the user interface. This includes functions
navigation tools for hypertexts
controls for playing audio and video
saving the user preferences
mouse button functionality
function key allocation
The number and variety of the controls are limited to
Control elements are always in the same place and have
the same appearance throughout the entire application. Inactivated controls
remain visible, but definitely can be detected as inactivated (e.g. dimmed
or grayed out).
Control elements of audio-visual presentations can be
intuitively operated (tape deck metaphor).
Clickable items can be easily recognized.
2.4.3. Screen Layout
Screen layout is clear.
In the standardized areas of the user interface the
same type of information is always found.
Too many concurrently opened windows are to be avoided.
2.4.4. Color Design
The color design of text backgrounds, entry forms and
control elements is discrete and unobtrusive. They do not impair the legibility
of text and the usability of picture information.
Color is used economically and never as an exclusive
In consideration of color-blind users color combinations
like red/green or blue/violet, in particular for texts and symbols, are
to be avoided. Instead of mixed colors, which only differ in nuance, clearly
differentiated colors are used. Framing a colored field with a black line
for the reinforcement of the contrast is recommended.
Color symbolism is consistent.
2.4.5. Help System
Operating logic and control elements are described in
an on-line help system.
Complete documentation is also available online or retrievable
from the application.
Help texts can be printed out.
Help texts are hyperlinked, context sensitive and provided
with an index.
Since on-line help systems are electronic publications,
the criteria specified in this catalog apply to them as well.
The presence of a printed manual is not necessarily
a quality criterion. However, where such a manual exists, it is understandable
and carefully laid out. The manual has a clear structure with a table of
contents and a glossary.
2.5. Dialog and Didactics
With EPMs, which are not pure tutorials, the learning
components are uniquely distinguished from the other sections of the publication.
Learning tasks are clearly outlined detailing their
content and an estimated learning time.
Teaching subjects are arranged in proper modules.
Learning objectives are specified for each module.
2.5.1. Embedding of the Knowledge
Since the knowledge to be acquired attains its meaning
only by integration with known facts, the basic concepts, on which the
learning process is based, are available through a hypertext environment.
As the progress of learning is influenced by the structure
of knowledge representation, the structure of chapters, pages and paragraphs
of hypertext documents are presented through an ordering principle as easily
comprehensible to the learner as possible. Often a book metaphor for the
presentation of textual knowledge is useful and recommended.
Learning strategies differ because they depend on the
prior knowledge of the learner, individual learner preferences and goals.
Taking this into account, the same content is accessible in different ways
from within an application.
2.5.2. Knowledge Assessment
Learning dialogs are not limited to closed questions
(as in multiple choice questions, the correct marking of objects),
but also contain open-ended questions (free text).
The analysis of free text input is robust, tolerant
concerning spelling and also supports synonyms.
The evaluation of answers to open-ended questions works
The evaluation of user responses is constructive, i.e.
an explanation is offered, rather than negative.
Learning dialogs are oriented in concrete, realistic
medical examination situations and use professional practice standards.
Typed in text can be corrected before it is analyzed
by the system (first edit, then send).
Simulations represent a real situation as realistically
as possible, focusing on the noticeable elements and processes and, if
necessary, are supported by graphical or photographic data.
Visualizations and close-to-reality simulations are
created in a way that they can be understood at the experience level of
Receptive sequences are connected to a special problem
and are able to be worked on after being played.
Problem solving tasks embedded into the learning process
are suitable to the topic and thus solidify as well as reinforce the newly
Concrete examples and case studies facilitate the learning
of new concepts.
Knowledge assessment integrated into the learning environment
gives the learner, on the one hand, feedback on the learning progress and,
on the other, permits user a measure of control of the learning path.
The technique of knowledge assessment is adapted to
the overall character of the system and, if necessary, goes beyond purely
text-based questions. For example, if interactive techniques such
as simulations are applied for instructional use, then such techniques
are used in the knowledge examination as well.
Examination of the acquired knowledge is oriented to
real assessment modalities and test sequences.
Assessment methods consider the following aspects:
meaningful formulation of distractors in multiple choice
4 to 8 answer alternatives in multiple choice questions
progressive rate of hints to the correct solution
intelligent presentation of questions also dependent
on the past answer behavior
question presentations also appear randomly as "quiz's"
possibility to get the correct solution
textual and graphical feedback on the learning progress
after finishing the individual learning units a evaluation
summary of the session is available.
2.5.3. Dialog / Navigation
The orientation within the learning path is possible
at any time.
Completed sections or modules are marked accordingly.
Training sessions can be interrupted, resumed or aborted
at any time. After an interruption the point of reentry is easily accessible.
The degree of user-control depends on the target group.
For beginners a "guided tour" requiring little user-control may be offered.
The degree of user-control also depends on the application's
content. In training systems, where the comprehension of the contents requires
a strict logical progression, the degree of user-control do not lead the
The possibility to adjust the amount of user-control
from a beginner's level upwards should be part of the program design, depending
on the learner's capabilities.
Dramatic elements are used as motivators (embedding
the contents in a story, simulation, role-playing, suspense elements, cartoons,
humor, rhetorical questions). The selection of these elements takes place
according to the communication behavior of the target group.
The system allows partial modification of the contents
enabling users to add their own information (e. g. cases).
ADGA Systems International (1996):
Instruction and Design Principles for Multimedia Computer Based Training.
AKAB (Arbeitskreis Automobilindustrie
Bildung) (1997). CBT-Kriterienkatalog. Version 1.00b.
Alessi S, Trollip SR (1991).
Computer-based instruction: methods and development. 2nd edition. Englewood
Cliffs, NJ: Prentice Hall.
Apple Computer (1992). Macintosh
Human Interface Guidelines. New York: Addison-Wesley. http://devworld.apple.com/dev/techsupport/insidemac/HIGuidelines/
Auhuber TC (1998). Entwicklung
und Evaluation eines computergestützten Lernsystems in der Medizin.
MicroPat - ein interaktiver Atlas der Histopathologie mit adaptierbarem
Tutor. Europäische Hochschulschriften VII/D/31. Frankfurt am Main
- Berlin - Bern - New York - Paris - Wien: Lang.
Barth A (1995). Elektronische
Information und Publikation: Strategie und Konzepte aus der Perspektive
des FIZ Karlsruhe. http://www.vchgroup.de/
Benyon D, Stone D, Woodroffe
M (1997): Experience with Developing Multimedia Coursware for the World
Wide Web: The Need for better Tools and clear Pedagogy. International Journal
for Human Computer Studies. 47: 197 - 218.
Bonsiepe G (1996). Der Designer
im Netz. http://www.ds.fh-koeln.de/~bonsiepe/desnetz.html
Euler D (1992). Didaktik des
computerunterstützten Lernens: praktische Gestaltung und theoretische
Grundlagen. 1. Auflage. Reihe: Multimediales Lernen in der Berufsausbildung.
Band 3. Nürnberg: BW, Bildung und Wissen.
Gräber W (1990). Das Instrument
MEDA. Ein Verfahren zur Beschreibung, Analyse und Bewertung von Lernprogrammen.
Kiel: Institut für die Pädagogik der Naturwissenschaften (IPN).
Herczeg M (1994). Software-Ergonomie:
Grundlagen der Mensch-Computer-Kommunikation. Bonn: Addison-Wesley.
ISO 9241 1998 (1998). Ergonomic
Requirements for Office Work with Visual Display Terminals, Part 10 to
17, International Standard.
ISO 9421 (1989). International
Standardisation Organisation, TC159, SC4, WG6. Ergonomic Dialogue Design
Criteria Proposal. Contribution for ISO 9421, Part 1.
Issing L (1997). Information
und Lernen mit Multimedia. 2., überarbeitete Auflage. Weinheim: Beltz.
Karrer U (1989). Computer-assisted
learning: Toward the Development and Use of Quality Course-ware. Bern,
Frankfurt am Main, New York, Paris: Lang.
Klar R, Bayer U (1990). Computer-assisted
teaching and learning in medicine. International Journal of Biomedical
Computing. 26: 7 - 27.
Leven FJ, Schulz S, Alle W,
Klar R (1995). Rechnergestützte Lehr- und Lernsysteme in den Klinika:
Stand und zukünftige Entwicklungen. In: Buchholz W, Haux R (eds.):
Informationsverarbeitung in den Universitätsklinika Baden-Württembergs.
Heidelberg. 187 - 193.
Microsoft Corporation (1995).
Windows Interface Guidelines for Software Design. The Microsoft Guidelines
for designing a user interface for Windows-based applications. Redmond,
Washington: Microsoft Press.
Multimedia in Manufacturing
Education Laboratory (1997). Multimedia Development Tools. http://mime1.marc.gatech.edu/MM_Tools/
Nicole, A (1990). Interfaces
for Learning: What do good Teachers Know that we don't? In: Brenda L (ed.):
The Art of Human-Computer Interface Design. Reading, Mass: Addison-Wesley.
113 - 122.
Open Software Foundation (1993).
OSF / Motif Style Guide (Revision 1.2). Englewood Cliffs, NJ: Prentice
Petersson G (1996). MEDCAL Review.
Riehm U (1992). Elektronisches
Publizieren: eine kritische Bestandsaufnahme. Berlin, Heidelberg: Springer.
Rogers Y, Scaife M (1997): How
can Interactive Multimedia facilitate Learning? In: Lee J (ed.): Intelligence
and Multimodality in Multimedia Interfaces: Research & Applications.
AAAZ Press, CA.
Schulmeister R (1996). Grundlagen
hypermedialer Lernsysteme. Theorie - Didaktik - Design. Bonn: Addison-Wesley.
Selbmann HK (1990). Validierung
von Software und Teachware. In: Baur M, Michaelis J (eds.): Computer in
der Ärzteausbildung. München: Oldenbourg. 165-173.
Shneiderman, B (1997). Designing
the User Interface - Strategies for Effective Human-Computer Interaction.
Reading, Mass: Addison-Wesley.
Thomé D (1989). Kriterien
zur Bewertung von Lernsoftware. Mit einer exemplarischen Beurteilung von
Deutsch-Lernprogrammen. Heidelberg: Hüthig.
Tognazzini B (1997). Tog on
Interface. Apple Computer, Inc.
Weinschenk S (1997). GUI design
essentials for Windows 95, Windows 3.1, World Wide Web. New York: John
Wiley & Sons.
Williams R (1994). The Non-Designer's
Design Book. Reading, Mass: Addison-Wesley.
|We wish to thank
Henriette Beran (Vienna), Gui Bonsiepe (Cologne),
Christoph Daetwyler (Bern), Vera Dammann (Giessen), Johannes Dietrich (Munich),
Florian Eitel (Munich), Michael Esser (Eppelheim), Josef Ingenerf (Lübeck),
Georg Koch (Freiburg), Ralf Köster (Göttingen), Göran Petersson
(Lund), Tina Reinhard (Würzburg), Michael Schmidts (Vienna), Richard
for important suggestions, ideas, material and criticism,
Matthias Holzer (Munich), Harold C. Lyon, Jr. (Munich),
Stefan Schlachter (Freiburg), Arthur Tesche (Munich)
for the translation.