Current possibilities of evidence-based speech therapy, in practice and
in clinical education: do we wish for more?
Hanneke Kalf, senior speech
therapist
University Medical Centre
Nijmegen, The Netherlands. E-mail: h.kalf@neuro.umcn.nl
In the Netherlands speech therapy as a profession hardly has its own academic history, because speech therapy is currently still a bachelor level education. Although this training formally demands that a speech therapist is able to read and use scientific papers (EBP recently became part of our professional profile), this is hardly ever put into practice. But the demands in our health care system are changing. Therefore evidence-based speech therapy in the Netherlands is a more or less ‘tradition-free’ but at the same time inevitable development. This speech will report about our first experiences with EBP-training for speech therapists and SLT-students.
In the Dutch system of
higher professional education the majority of speech therapists receive four
years of training to become a therapist (bachelor level). That means among
other things that they only have mastered some basis statistics. A small
minority chooses an additional academic training and receives also a master’s
degree. Scientific studies in the field of speech therapy are carried out by
some scientifically skilled speech therapists, but mainly by linguists
(language disorders, aphasia), psychologists (stuttering) or doctors (hearing,
dysphagia) etc. Also, our national logopedic journal Logopedie en Foniatrie is not a scientific journal. As a result of
that the focus in educating EBP is on using
scientific evidence, which is in fact the meaning of evidence-based practice. Another aim is to narrow the
gap between scientists and practitioners, because the best quality of therapy
can only be provided to patients and clients by the therapists themselves.
In 2001 we (Joost de Beer
and Hanneke Kalf) by accident initiated the same idea of developing courses
about evidence-based practice for speech therapy. In 2002 we started giving
courses, in which we tried to make the philosophy and tools of ‘Evidence-based
Medicine’ from the McMaster University in Canada and the Dutch Cochrane Centre
useful for speech therapists. In our view EBP for speech therapy is based upon
the following ideas:
-
It is necessary as a speech
therapist to be aware of the true validity of the diagnostic and therapeutic
interventions that are professionally used.
-
A speech therapist should
be able to find, interpret and use scientific evidence, also as a
practitioner-not-trained-to-be-a scientist or, when limited in time and
facilities, be able to ask adequate clinical questions.
-
The original aim of
evidence-based practice: “the integration of best research evidence with
clinical expertise and patient values”, it is also client-orientated, nor rules
out the expertise of the individual therapist.
The course has been
designed and put together by a scientifically skilled linguist (JdB) and a
speech therapist trained in evidence-based practice (HK), thus combining
scientific experience and the need for a more evidence-based way of delivering
therapy from the profession itself. Interestingly, although physicians receive
an academic training, all over the world short EBM courses are provided for
doctors. Our course is partly based on the 3 day short courses in ‘evidence
based medicine’ for physicians. We wanted the course to be accessible for the
average speech therapist and designed a programme of 3 evenings (or afternoons)
of 3.5 hours and almost the same amount of self-study, in total 20 hours. The
course is aimed at developing two skills:
-
being able to search for
scientific evidence in relevant databases
-
being able to critically
appraise an article about the validity of a diagnostic tool and an efficacy
study or randomised controlled trial.
For the critical appraisal
of articles we use a standardized form, drafted by several Dutch scientific
institutes, e.g. the Dutch Cochrane Centre, for training courses on evidence
based medicine.
Another characteristic of
this course is the small number of students per lecturer, that is 10 to 12,
because we anticipated that individual attention during acquiring both skills
(searching and appraising & calculating) would be necessary for efficient
learning. Until our own course book is being published we use ‘Inleiding in
evidence-based medicine’ by Offringa et al. (2000) for the theoretical part,
the Dutch equivalent of the EBM manual by Sackett and colleagues (2000).
Still, the main target of
the EBP programme for speech therapists is a change of attitude: being more
critical about the validity and value of published articles, also in Logopedie en Foniatrie, asking for
evidence in clinical courses, and trying to search for evidence in a database
whenever a clinical question comes up.
Some aims
According to Sackett et al.
(2000), when appraising an article three main questions should be answered:
1. Are the results of this study (about efficacy etc.)
valid?
2. Are these valid results important?
3. Are the valid and important results applicable to my
patients?
(These criteria and several
toolkits explaining the criteria and calculations can also be found on various
sites on the Internet, for example on www.nettingtheevidence.org.uk
or www.cebm.net or www.ebmn.nl).
When adjusting
the philosophy of EBM into EBP training for speech therapists we encountered
among other things the following.
In our experience it is indeed possible to skip the statistics paragraph in scientific papers in order to calculate the magnitude of the results (after concluding that the study results were valid), based on a 2 x 2 table and using only a simple palm calculator.
Diagnostic studies useful for EBP appraisal are validation studies of a new screening instrument. When evaluating a diagnostic study, the first main issues are the reference test or gold standard and blinding of the comparison with the index test. Some therapists realise for the first time the risks of invalid tests through false positive and false negative findings. The most difficult part is the concept that the sensitivity and specificity of a test or screening represent the diagnostic value, but that the clinical diagnostic relevance of a screening is demonstrated by the positive and negative predictive value, depending on the prevalence of the disorders. An example: the sensitivity and specificity of the language screening for pre-school children by Laing et al. (2002) is 66% and 89% respectively, giving a positive likelihood ratio of 6.2 and a negative likelihood ratio of 0.4. What does that signify: that due to the moderate sensitivity too many children are being missed? On the contrary: with the given prevalence (or pre-test probability) of the 23%, the negative predictive value is 90% and the positive predictive value is 65% (post-test probabilities). Meaning that only 10% of the language delayed children is being missed by this screening, but that the screening generates 35% false positives and thus one-third unnecessary referrals. It is important to realise that low positive predictive values usually are inherent to low pre-test probabilities.
In evidence based medicine (Sackett et al., 2000; Offringa et al., 2000) the magnitude of the therapy effect is determined by calculating risk reductions (RR, ARR, RRR) and numbers needed to treat (NNT). In efficacy studies this is only possible when the provided data concern the numbers of patients who had benefit of the treatment and who had not. For that matter, in speech therapy we do not aim at ‘risk reduction’ but at ‘benefit increase’ for a patient. The example is from the study of the effectiveness of training volunteers as conversation partners (using ‘Supported conversation for adults with aphasia’ – SCA) by Kagan et al. (2001). After approving the validity of the study one of the outcomes is the transaction of information by the participant with aphasia with help of a trained volunteer. In the experimental group in 17 of the 20 cases (85%) the transaction by participants with aphasia after training of volunteers was better, in 3 cases it was the same of worse. In the control group in only 5 of the 20 cases (25%) the transaction was better. The absolute benefit increase (ABI) is 85-25 = 60%, meaning that when training 100 volunteers in 60% of the cases the transaction of participants with aphasia will be better. For 15% of the volunteers training does not work and 25% of the volunteers does not need a training to be an adequate conversation partner. That equals an NNT of 2 (1.7 = 1 / ABI), meaning that almost 2 volunteers have to be trained in order to have one successful conversation partner for participants with aphasia. Another way of interpreting the results is the benefit increase or benefit ratio: the ratio of the benefit chances with therapy and without therapy. In this example this is 85%/25% = 3.4, meaning that after this training the chances of better transaction improve with 3.4 times. These results are not given in the article, but are easily be calculated with the provided data. The authors state that a chi square analysis indicated that the differences between the two groups were statistically significant and they also presented several other statistical calculations of the results. But according to EBP in fact the only ones needed to demonstrate the magnitude of the effect are the ones mentioned here. Of course it is important in efficacy studies to know whether the differences between the experimental group and the control group are statistically significant, but when the difference is significant does that mean that it is clinically relevant and that every patient benefits from the treatment? Usually a number of patients improve without the therapy due to spontaneous recovery or natural growth and some do not improve despite the therapy. In EBP the question is: how many? What are the chances of recovery due to the therapy, because it is hardly ever 100%? Whether the result is large enough to conclude that the therapy should be implemented is another decision, e.g. dependent of the chosen outcome, intensity of the treatment etcetera.
But are there enough studies in the field of speech therapy worth appraising? Are the results in SLT studies presented in a way that clinical relevance can be calculated? Does EBP also have consequences for researchers?
After one and a half-year
we have learned that usually course members indeed have little or no experience
with scientific databases or with reading and using scientific papers. A small
but time taking threshold is that students are forced to search and read in English
instead of Dutch. The large databases like Medline,
Cinahl, PsycInfo and Cochrane all
are in English, as are most scientific studies. Currently we have trained about
100 speech therapists, mainly lecturers who are responsible for the initial
training at one of the 7 universities of professional education, and speech
therapists that work in an academic or university hospital. But also some
speech therapists who work in a private practice or in schools. In general the
speech therapists that have done this introduction training report that they
have gained valuable skills, that much more colleagues should do the course and
that EBP should be part of the initial training. At the same time they feel the
need for much more exercise and practice to become really familiar with it.
According to the latest evidence, intensive 3 day courses on EBM for doctors
leads ‘to a clinically meaningful improvement of knowledge and skills’
(Fritsche et al., 2002), a promising development.
In 2001 and in 2002 several
lecturers responsible for the initial training of speech therapy students first
started to familiarize logopedic students with the principles of EBP. One of
the experiences is that third year students can adequately be trained in EBP,
comparable to the way in which graduated speech therapists are trained. For
example the obligation to search for non-Dutch scientific articles is not a
serious threshold. Currently the debate is about the way in which an EBP
programme can be implemented in the curriculum in alignment with the existing
elements of methodology, basic statistics etcetera.
Discussion
Finally some practical
points and limitations. Access to the Internet is compulsory when you need to
search for evidence in electronic databases. Looking for a very specific answer,
but also when there are many papers available can take a lot of time. Only
PubMed and Cochrane are free databases on the Internet. Besides Medline,
important databases like Cinahl and PsycInfo are only available by subscription
or in university libraries. And than only the abstracts are found. Many of the
journals, except the free journals like the British
Medical Journal, are also only available by subscription or in university
libraries. And when articles have to be ordered in other libraries there is
also a financial aspect. So, only the speech therapists that are provided with
enough time and facilities are able to uncover useful evidence?
Another issue is language.
The main databases in our experience are American (Medline, Cinahl, PsycInfo[1])
and the non-English scientific journals in these databases are a minority. Is
that a correct reflection of the international academic community, is English
considered to be the scientific language, or are scientific journals in French,
German or Italian being missed? Anyway, in the Netherlands English is the most
used second language, so a publication-bias is inevitable.
During the last decade also
attempts were made to summarize and interpret evidence into reviews and
guidelines. In the Cochrane database several reviews about logopedic questions
can be found, but the lack of good evidence (RCT’s) frequently leads to
inconclusive answers. Also other kinds of overviews were published, for example
Does speech therapy work? by Enderby & Emerson in 1995 or the ‘Supplements on
Treatment Efficacy’ in the Journal of
Speech, Language and Hearing Research in 1996 and 1998. Also national
logopedic associations like the RCSLT and NVLF are currently trying to get
relevant evidence available in guidelines for all speech therapists. It is important to have national and international (!)
statements about best practice in order to leave bad practices behind. But
guidelines become out of date and in our opinion it is still worthwhile for
every speech therapist to have some basic skills in searching and interpreting
scientific studies and to be able to use or reject scientific results with
valid arguments.
In conclusion, in the Netherlands evidence-based practice is a new development for speech therapist and still only a small number of speech therapists have attended to EBP courses. Although EBP-courses are not about statistics or SPSS it is considered difficult and time-consuming.
And we don not know whether
short courses are enough to achieve the main goal: change of attitude. The
debate in the medical community about the relevance of EBM courses is
promising, but much of this debate is only beginning to reach our profession,
e.g. at this CPLOL conference. On the other hand the first course book
specifically discussing the possibilities of EBP for speech therapy by
Australian speech pathologists has been announced (Reilly et al., in press).
Enderby P, Emerson J. Does speech and language therapy work?
London: Whurr Publishers, 1995.
Fritsche L, Greenhalgh T,
Falck-Ytter Y, Neumayer H-H, Kunz R. Do short course in evidence based medicine
improve knowlegde and skills? Validation of Berlin questionnaire and before and
after study of courses in evidence based medicine. BMJ, 2002, 325, 1338-1341.
Kagan A, Black SE, Duchan
JF, Simmons-Mackie N, Square P. Training volunteers as conversation partners
using ‘Supported Conversation for Adults with Aphasia’ (SCA): a controlled
trial. Journal of Speech, Language, and
Hearing Research, 2001, 44, 624-638.
Laing GJ, Law J, Levin A, Logan S. Evaluation of a structured test and a parent led method for screening for speech and language problems: prospective population based study. BMJ, 2002, 325, 1152-1156.
Offringa M, Assendelft WJJ, Scholten RJPM. Inleiding
in evidence-based medicine. Klinisch
handelen gebaseerd op bewijsmateriaal. Houten: Bohn Stafleu Van
Loghum, 2000.
Reilly S, Perry A, Oates J,
Douglas J. Evidence Based Practice in
Speech Pathology. London: Whurr Publishers, in press.
Sackett DL, Strauss SE,
Richardson WS, Rosenberg W, Haynes RB. Evidence-based
Medicine. How to practice and teach EBM. Edinburgh: Churchill Livingstone,
2000.
http://sumsearch.uthscsa.edu/cgi-bin/sumsearch.exe
www.cochrane.nl
www.ncbi.nlm.nih.gov/entrez/query.fcgi
[1] PsycInfo is the abstract database of the American
Psychological Association and has 1,830 journals indexed, also about
neuroscience etc. Cinahl (Cumulative Index of Nursing and Allied Health
Literature) contains 2,286 journals mainly about nursing, but also 36 journals
on speech language pathology and audiology.