Variation in pragmatic performance in children with language disorders: establishing baselines for an intervention study
Catherine
Adams and Julian Lloyd
Human
Communication and Deafness Group,
University
of Manchester, UK
ABSTRACT
This paper reports the preliminary phase of a project aimed
at establishing appropriate outcome measures for intervention with children who
have pragmatic language impairments (PLI). Specifically this stage aimed to
ascertain the degree of variability on pragmatic assessments such as
conversational coding and elicitation tasks. 15 children with PLI (mean age 9;5
years) and an age-matched control group were assessed on two occasions with a
six-week interval on a speech act elicitation task and a conversation task. The
results gave an indication of the utility of assessment procedures in measuring
any change in pragmatic performance over time and indicate what the range of
natural variation might be. Implications for the design of intervention
effectiveness studies for children with PLI are considered.
INTRODUCTION
In the population of children with developmental language
disorders, a subgroup has been identified whose difficulties lie principally
with the use of language in social contexts. Bishop (2000) has termed this
subgroup “Pragmatic Language Impairment” (PLI) but it has may be more familiar
under its previous title of “semantic-pragmatic language disorder”. Children
who have PLI are typically verbose, have poor turn-taking skills, difficulty
staying on topic, and significant difficulty in developing conversational
skills. PLI is thought to be closely related to autism (Boucher, 1998) and
these children share many of the social and cognitive features of autistic
children, with the exception that their social withdrawal is less extreme.
Work on the nature of PLI is ongoing but speech and language therapists have to
manage the communication needs of these children with little support from the
research base. There is little existing evidence that pragmatic ability can be
enhanced by facilitation or by direct intervention and yet a significant
proportion of services in educational speech and language therapy is directed
to the amelioration of pragmatic difficulties. This paper reports findings from
the preliminary stage of a study that aims to determine whether the underlying
communication impairment in PLI can be facilitated by language intervention.
One of the reasons for the lack of an evidence base for
these interventions is the difficulty of “measuring” pragmatic behaviour in a
reliable way. Often assessment of pragmatics is carried out by an
impressionistic checklist approach as the only feasible option for the busy
practitioner (Adams, 2002; Paul, 1995; Perkins, 2003; Prutting & Kirchner,
1987). Though such instruments have great value in structuring observations for
planning interventions and contextual support for these children, they are of
limited value when we wish to prove that change can be made as a result of
intervention. The instruments which are required need therefore to be sensitive
to the particular nature of the problems of children with PLI and sensitive to
change, while at the same time demonstrating reliability.
Another problem for pragmatic “measurement” is the natural variation in
communication performance from one time to another. Pragmatics is not an aspect
of communication that can be carefully elicited, structured or quantified in
the way that formal tests of vocabulary, say, can be. Therefore, to set up an
appropriate regime for evaluating changes in pragmatic performance as a result
of an intervention, it is necessary to have factored in the “allowable”
variation in pragmatic ability on an assessment on different occasions. If this
is not done, there is a danger of showing significant changes which might have
arisen simply as a result of some typical variation in the unstructured
assessment procedures.
This paper considers two assessment tools that were developed to assess
pragmatic behaviour. (1) a scripted procedure for eliciting speech acts and
other types of pragmatic behaviour and (2) the Analysis of Language Impaired
Children’s Conversation (ALICC), a system of conversation sampling and analysis
which can measure change in pragmatic behaviour, such as initiation,
responsiveness, conversational repairs and topic switching (Adams & Bishop,
1989; Bishop, Chan, Adams, Hartley & Weir, 2000).
METHOD
Subjects and
assessment schedule
15 children (10 boys and 5 girls, mean age = 9;5 years;
S.D.= 2;5; range = 7;3 - 11;6).) identified as having PLI by their speech and
language therapists were included in the study. Diagnostic information was
supplemented by data from completed Children’s Communication Checklists
(Bishop, 1998). An age-matched control group (mean age = 9;4 years; S.D.=1;5;
range=7;1-11;0) also took part in the study. All children participated in the
elicitation task. A subset of ten
children from the PLI group (6 boys and 4 girls) with a mean age of 9;10 years
(S.D.=9;10; range=7;11 to 11;6), and 10 children from the control group with a
mean age of 9;8 years (S.D.= 1;3; range=7;3 to 11;0) also took part in the
conversation study.
Assessment on elicitation and conversation tasks took place
in a quiet room with only the child and tester present. The sessions were
videoed and a protocol was developed to score the recordings for the
elicitation task. Conversational coding was completed from the video using the
procedure described below. To permit an assessment of variation in pragmatic
performance, each child was assessed on a second occasion. A two-week interval
between Time 1 (T1) and Time 2 (T2) was considered the optimum interval, but
due to school access problems the period between testing ranged from 2-16
weeks.
Elicitation
procedure
A scripted procedure for eliciting pragmatic behaviours was
developed based on the work of Creaghead. Creaghead (1984) put forward a
“communicative-demand” strategy (known at the time as the “Peanut Butter
Protocol”) for eliciting and assessing pragmatic behaviours (Paul, 1995). This
approach involves setting up situations that require the child to exhibit
certain behaviours if they are to behave appropriately in that situation. The
procedure included a series of tasks and props that can be used to elicit
particular communicative behaviours. In an attempt to elicit a request for an object, for example, a
child was asked to get a pen from a locked box. A request for a key was
therefore appropriate behaviour in that situation. These tasks and props
included a map task to elicit clarification requests and responses to
clarification requests (see Lloyd, 2003); toy telephones to elicit greetings,
closings, questions and answers; a magic trick to elicit hypothesising,
explaining and predictions; and fake biscuits to elicit choices and comments.
(See Appendix).
For each type of behaviour that was examined the scoring was
as follows:
- yes
– the behaviour was elicited
- no
– the behaviour was not elicited
- not
observable – the behaviour could not be observed due to contextual factors
(e.g., child became uncooperative, or one of the prompts did not work
properly).
Conversation
Task
The subjects were assessed through a conversation task that
was based on the Analysis of Language Impaired Children’s Conversation (ALICC),
developed by Bishop, Adams and their colleagues (Adams and Bishop, 1989; Bishop
& Adams, 1989; Bishop, Hartley and
Weir, 1994; Bishop et al, 2000; Adams, Green, Gilchrist and Cox, 2002).
Conversations were gathered using the semi-structured method devised by Adams
and Bishop (1989). This method uses black and white photographs as prompts to
initiate particular topics:
- T1
– a visit by the doctor; a birthday party; a car breaking down
- T2
– building a bonfire; a family day trip; taking a pram from a rubbish tip
The aim was to use the photograph to initiate conversation
with the child but then to move on to discuss the child’s own similar
experiences. The recordings were transcribed, and turns and utterances were
identified and numbered using the guidelines presented by Bishop et al. (2000).
A particular focus of attention was adult solicitations: utterances by the
adult that solicit either information
or acknowledgement from the child.
Solicitations for information include wh-questions (“Where did you go on
holiday?”), yes-no questions (“Do you like chocolate?”), and clarification
requests (C: “I went with Paul”; A: “Who’s Paul?”).
Every transcribed conversation was coded for communicative
acts using utterance as the unit of analysis. Then, using Adams et al. (2002)
as a basis, the following indices of conversational behaviour were employed:
- discourse participation;
- conversational dominance;
- assertiveness;
- verbosity;
- responsiveness;
- meshing.
Discourse participation is the ratio of child utterances to
tester utterances. It can be used to evaluate each partner’s contribution to
the conversational floor time. A ratio of 1.0 suggests that both partners have
contributed equally to the conversation. A ratio less than 1.0 suggests the
tester has dominated the conversation. A ratio greater than 1.0 suggests the
child has dominated the conversation (Lloyd, Lieven and Arnold, 2001).
Conversational
dominance is an index that can be used to estimate the degree to
which the child dominates the conversation by repeatedly requesting information
or by providing unsolicited information. The following formula was derived:
Conversational dominance = child [first
parts + statements]/adult + child[first
parts + statements]
Assertiveness is
an index that indicates the child’s tendency to initiate conversational
exchanges. The following formula was derived:
Assertiveness
= Child first parts/Total child utterances
An index of verbosity
was used to identify a tendency to verbosity in the form of extended
unsolicited talk in a single turn. It was defined as the number of child turns
that included four or more statements in sequence as a proportion of the total
turns for that conversation.
Verbosity = number of child turns which
contain 4 or more statements in sequence/total number of turns in conversation
Responsiveness
= proportion of child responses to
tester utterances ending in a soliciting or neutral intonation In this analysis nods and shakes of the head
were treated as yes/no answers.
Meshing
was defined as the quality of fit between adult
solicitations and child responses. Four categories of meshing have been defined
(see Bishop et al. (1998) for a more detailed description of these categories):
a) Adequate: The response is judged to be a good fit with
the first soliciting part (A: “where are you going on Saturday?; C: “to the zoo”).
b) Inadequate: The child produces a less than optimal
response because of some linguistic limitation, comprehension failure, or lack
of general knowledge. When using this code no allowance is made for age (e.g.,
A: “Where’s Tenby?”; C: “Long way away”).
c) Pragmatically inappropriate: The response does not fit
the social and/or communicative context of the soliciting part. This may be
because the child has failed to take into account previously given information,
ignores an adult solicitation, produces a tangential response, an over-literal
response that does not appreciate the speaker’s intention, or uses “don’t know”
uncooperatively (e.g., A: “have you ever been to the doctor?”; C: “I had an
apple a day”; A “where did your dad take you last Saturday?” C: “can you please
not ask that question”).
d) No response: The child is given an opportunity to respond
but does not do so.
An index of
problematic responses was derived using the following formula:
Response problems = child (inadequate responses + pragmatically
inappropriate responses + no responses)/ total child responses
An index of pragmatically
inappropriate responses was calculated as follows:
Pragmatic problems = pragmatically inappropriate
responses/total child responses
Results
Elicitation task
The elicitation procedure did not distinguish between the
PLI and control groups on the number of communicative acts elicited. Both
groups were performing at or close to ceiling level for most of the items,
rendering any more detailed presentation of results here trivial. Similarly,
little variation was found between T1 and T2 for either the PLI or the control
group. Though the results did suggest that knowledge gained in the task during
T1 could affect the performance of some children in T2 (for example, in T2 the
child already knows that a pen is hidden in the locked box and therefore does
not have to ask about the location of the pen), similar results were found for
both the PLI and control groups. Figure 1 shows a pattern of responses for one
item on the elicitation task (Requesting an Object) which is representative of
all other items. The majority of responses are correct (=Yes) with little variation between the groups and
little change over time and a pronounced ceiling effect. Children with PLI
tended to show a little more variability when the data is displayed graphically
but this is not significant for any items.
Figure 1: Distribution of elicitation codes for speech act
item number 26 “Requesting an Object”
from Time 1 and Time 2
Conversation
task
In the following analyses, a 2x2 mixed factorial ANOVA
design was employed. The various conversational measures described in the last
section were the dependent variables. The independent variables were time of
testing (2 levels: T1 and T2) and group (2 levels: PLI and Control). Time was
manipulated within subjects, while group was manipulated between subjects.
Table 1 shows the total and mean number of turns and
utterances for the PLI-tester and controls-tester conversations. No significant
main effect of time or group, and no significant interaction between time and
group, was found for any of these measures of turns and utterances. These
findings suggest no significant differences between the PLI-tester and Control-tester
conversations concerning the amount of talk that took place between the
participants.
Table 1. Total
and mean number of turns and utterances for the PLI-tester and Control-tester
conversations at Time 1 and Time 2
|
|
|
T1
|
|
|
T2
|
||
|
|
Total |
Mean |
SD |
|
Total |
Mean |
SD |
|
PLI
Conversations |
|
|
|
|
|
|
|
|
Turns |
1498 |
149.80 |
(54.37) |
|
1452 |
145.20 |
(51.24) |
|
Adult utterances |
1371 |
137.10 |
(46.66) |
|
1331 |
133.10 |
(36.99) |
|
Child utterances |
1212 |
121.20 |
(40.16) |
|
1172 |
117.20 |
(64.73) |
|
Discourse participation a |
- |
0.87 |
(0.29) |
|
- |
0.80 |
(0.27) |
|
|
|
|
|
|
|
|
|
|
Control
Conversations |
|
|
|
|
|
|
|
|
Turns |
1589 |
158.90 |
(59.01) |
|
1648 |
164.80 |
(81.54) |
|
Adult utterances |
1406 |
140.60 |
(31.08) |
|
1503 |
150.3 |
(46.69) |
|
Child utterances |
1407 |
140.7 |
(81.50) |
|
1468 |
146.80 |
(93.56) |
|
Discourse participation a |
- |
0.92 |
(0.36) |
|
- |
0.89 |
(0.33) |
a Uncodeable utterances were excluded
from the calculation of discourse participation ratios
Table 2 shows the mean proportions of the communicative act
types for the PLI and control groups at T1 and T2. No significant main effect
of time, and no significant interaction between time and group, was found for
any of the communicative act types. A
significant main effect of group was found for questions [F(1,18)=5.62, p=0.029],
clarification requests [F(1,18)=4.83,
p=0.041], clarifying [F(1,18)=6.29, p=0.022] and recapitulation [F(1,18)=5.31,
p=0.033]. The PLI group used higher
proportions of questions, clarification requests, clarifying responses and
recapitulations (recaps) than the control children.
Table 2. Mean
proportions of the communicative act types for the PLI and Control groups at
Time 1 and Time 2
|
|
|
|
PLI |
|
|
|
|
CON |
|
|
|
|
|
T1
|
T2
|
|
T1
|
T2
|
||||
|
Communicative
Acts |
|
Mean |
SD |
Mean |
SD |
|
Mean |
SD |
Mean |
SD |
|
Questions a |
|
2.08 |
(1.98) |
3.40 |
(3.47) |
|
0.78 |
(1.45) |
1.18 |
(1.48) |
|
Answers |
|
45.31 |
(8.67) |
46.14 |
(21.49) |
|
50.72 |
(19.52) |
46.51 |
(17.76) |
|
Statements |
|
18.99 |
(9.13) |
19.10 |
(16.80) |
|
24.36 |
(18.51) |
27.40 |
(18.77) |
|
Clarification requests a |
|
0.42 |
(0.75) |
0.60 |
(1.36) |
|
0 |
(0) |
0 |
(0) |
|
Clarifying a |
|
4.87 |
(2.67) |
4.05 |
(6.37) |
|
1.17 |
(1.37) |
1.58 |
(1.26) |
|
Denial/Correction |
|
0.33 |
(0.56) |
0.35 |
(0.77) |
|
0 |
(0) |
0 |
(0) |
|
Recapitulation a |
|
1.24 |
(1.97) |
1.65 |
(1.04) |
|
0.12 |
(0.28) |
0.64 |
(0.81) |
|
Evaluation |
|
0.07 |
(0.22) |
0.35 |
(0.77) |
|
0.03 |
(0.98) |
0.24 |
(0.49) |
|
Conversational mechanics |
|
16.27 |
(10.08) |
16.05 |
(8.59) |
|
14.40 |
(6.52) |
14.97 |
(5.92) |
|
Other |
|
0.97 |
(0.84) |
0.57 |
(0.66) |
|
0.12 |
(0.38) |
0.52 |
(1.42) |
|
Uncodable |
|
9.46 |
(4.22) |
7.59 |
(6.58) |
|
8.00 |
(3.48) |
6.96 |
(4.40) |
a
PLI>Con, p<0.05
Table 3 shows the mean conversational behaviour indices for
the PLI and control groups at T1 and T2. A significant main effect of group was
found for response problems [F(1,18)=13.954,
p=0.002] and pragmatic problems [F(1,18)=6.29, p=0.000], with the PLI group displaying more problematic responses
than the controls.
Table 3. Mean
conversational behaviour indices scores for the PLI and Control groups at Time
1 and Time 2
|
|
|
|
PLI |
|
|
|
|
CON |
|
|
|
|
|
T1
|
T2
|
|
T1
|
T2
|
||||
|
Communicative
Indices |
|
Mean
|
SD |
Mean |
SD |
|
Mean |
SD |
Mean |
SD |
|
Conversational dominance |
|
0.28 |
0.15 |
0.25 |
0.20 |
|
0.30 |
0.24 |
0.31 |
0.24 |
|
Assertiveness |
|
0.03 |
0.02 |
0.04 |
0.03 |
|
0.01 |
0.02 |
0.01 |
0.01 |
|
Verbosity |
|
0.02 |
0.02 |
0.01 |
0.02 |
|
0.03 |
0.02 |
0.02 |
0.02 |
|
Responsiveness |
|
0.95 |
0.04 |
0.93 |
0.05 |
|
0.95 |
0.28 |
0.93 |
0.08 |
|
Response problems a |
|
0.27 |
0.13 |
0.33 |
0.13 |
|
0.08 |
0.09 |
0.12 | |