Comparing the Impact of Two Different Designs for Online Discussion 369
3 Results
The online version of the volume will be available in LNCS Online. Members of
institutes subscribing to the Lecture Notes in Computer Science series have access to
all the pdfs of all the online publications. Non-subscribers can only read as far as the
abstracts. If they try to go beyond this point, they are automatically asked, whether
they would like to order the pdf, and are given instructions as to how to do so.
Please note that, if your email address is given in your paper, it will also be in-
cluded in the meta data of the online version.
Table 1. A Summary of Online Discussion Participations by the Students
_________________________________________________________________________
Fall 2007 Spring 2008
__________________________________ ____________________________________
Topics Accesses Messages Words Topics Accesses Messages Words
per per
message message
__________________________________ ____________________________________
Lesson 1
About me 477 32 83 High stakes test 714 32 200
Assessment Benefit of
topic 488 36 70 assess. know. 537 28 130
Lesson 2
Reliability 405 34 126 Test reliability 481 24 111
True score 449 33 54 True score 523 17 215
Lesson 3
Validity Test
example 346 31 113 validity 717 35 101
Test Test
validation 363 30 116 validation 490 25 95
Lesson 4
Test bias 354 32 107 Biased assess. 344 28 140

Test Test
accommod. 240 28 99 accommod. 415 35 117
Lesson 5
Cognitive Cognitive
Level 306 28 76 level 522 30 146
Affective/ Affective/
Psychomotor 278 26 93 psychomotor 564 30 121
Lesson 6
Guideline Guideline
for item for selected
construction 201 26 113 response item 314 25 125
Type of test Selected
item 175 27 106 response item 314 26 106
________________________________________________________________________
Average 340 30 96 Average 495 28 134
370 Y. Yao
3.1 Student Participation
A summary of quantitative data regarding student participation in the online discussions
is provided in Table 1. The data shows that the students in the online course visited each
forum on an average of 340 times in Fall 2007, and 495 times in Spring 2008, an
increase of 46% over the previous semester. There was an average of 30 messages per
discussion topic provided by students in Fall 2007, and about the same number (an
average of 28) of student messages in Spring 2008. The average number of words a
student used in a posted message was 96 in the fall semester, and 134 in the spring, an
increase of 40% over the previous semester.
Table 2. A Summary of Online Discussion Participations by the Instructor
__________________________________________________________________________
Fall 2007 Spring 2008
___________________________________ ____________________________________
Topics Accesses Messages Words Topics Accesses Messages Words

per per
message message
___________________________________ ____________________________________
Lesson 1
About me 66 6 34 High stakes test 72 4 100
Assessment Benefit of
topic 145 21 14 assess. know. 43 2 124
Lesson 2
Reliability 184 24 11 Test reliability 22 1 373
True score 196 24 10 True score 32 3 95
Lesson 3
Validity Test
example 119 19 34 validity 20 1 145
Test Test
validation 194 23 32 validation 16 1 424
Lesson 4
Test bias 96 18 28 Biased assess. 14 1 411
Test Test
accommod. 85 20 16 accommod. 13 1 460
Lesson 5
Cognitive Cognitive
Level 121 22 15 level 29 2 169
Affective/ Affective/
psychomotor 122 21 12 psychomotor 30 2 221
Lesson 6
Guideline Guideline
for item for selected
construction 73 21 12 response item 6 1 244
Type of test Selected
item 66 21 19 response item 5 1 239

__________________________________________________________________________
Average 122 20 20 Average 25 2 250
Comparing the Impact of Two Different Designs for Online Discussion 371
3.2 Instructor Involvement
The involvement of the instructor in terms of the number of visits to the discussion
forums, the number of messages, and the number of words per message is summa-
rized in Table 2. The instructor visited each forum on an average of 122 times in Fall
2007 and 25 times in Spring 2008. Given that there were 24 students in each semester,
the instructor visited each student’s postings about five times in the Fall semester, and
only once per student in the Spring semester. The average number of messages posted
by the instructor for each discussion topic also dropped significantly, from 20 in the
Fall to 2 in the spring. On the other hand, the average number of words used by the
instructor per message increased drastically, from an average of 20 words per mes-
sage in the Fall 2007 semester, to an average of 250 words per message in the Spring
2008 semester. The sharp increase in the words reflected the fact that the instructor
used relatively brief comments to each student in the Fall semester and much detailed
feedback at the end of the discussions in the Spring semester. The total number of
words the instructor used for each discussion forum averaged 400 for the Fall semes-
ter and 500 for the Spring semester.
3.3 Student Interactivity
Aside from the quantitative data, the content of the discussion forums was examined
to compare the potential impact of the discussion designs. One difference that was
relatively easy to identify was the extent of interactivity between the students. Al-
though there was some interactivity in the Fall semester, it was only discernible in 5
out of the 30 messages students posted in response to the topic of test validation.
Overall, the postings in that semester’s discussion board read more like a dialogue
between each individual student and the instructor, with each student responding to
the message originally posted by the instructor, and the instructor following the re-
sponse with a brief comment on the merit of the student’s response. In contrast, in the
Spring semester, each student was required to respond to the previous post. It was

evident from the postings that each student had read at least the previous post before
providing any response in the forum. Many messages included acknowledgment of
several postings that were provided earlier.
3.4 Opportunities for Questions
Related to the difference of interactivity is the opportunity for students to raise mean-
ingful questions. In the Spring section, there was sufficient evidence of students
asking their fellow students to clarify their positions or challenge them to a new per-
spective. For instance, in the discussion forum on the topic of test accommodation, a
student mentioned a friend of hers who needed accommodation due to dyslexia. She
observed that test accommodations were important, but should be only provided on a
case by case basis. Another student followed up by drawing attention to potential
issues with test accommodations, even when the provision of accommodations was
justified in this case:

372 Y. Yao
Did she say whether or not the reader’s voice intonation, eye contact, or
body language ever helped her get the right answer, or maybe even mis-
guided her to a wrong answer? I ask this because I would be concerned about
an added advantage or disadvantage to the tester.

In another post, a student questioned the benefit of accommodating students in
testing over the long term:

I have questions about whether or not students benefit in the long run by hav-
ing exams read to them when they are poor readers… I question the wisdom
in constantly reading exams to students. Students will learn that they have no
need to try harder…

There were five instances where such thought provoking questions were asked re-
garding test accommodation in the Spring section. In contrast, no such question was

found in a corresponding forum in the Fall section. Instead, most of the messages
posted in the semester were attempts to answer the instructor’s original question
without raising any further questions for either the instructor or the rest of the class.

3.5 Evidence of New Learning
Not only was there a difference in the opportunity to raise thoughtful questions, there
was also a difference in the opportunity for students to engage in meaningful inquires
that would provide for them insights or new perspectives. Such opportunities were
rare in the discussions in the Fall semester, where the conversations were mostly
restricted to those between the instructor and an individual student. A typical response
by the instructor to a student’s post on the topic of test validation was: “I think that
would be a great idea to validate a test, by giving it to students in another grade!”
Although the instructor’s comments were often worded in a constructive way, seldom
did the conversation between the instructor and the student proceed much further.
Occasionally, a conversation might occur among the students themselves in the
Fall section. For instance, the following is a message pertaining to the topic of test
validation:

Since I am the only teacher for my area of instruction, it makes it difficult to
have another teacher review the exam; however, your post got me thinking
that there are a few other instructors in the local area, some with many years
[of] experience and they could be a great resource in developing my pro-
gram. Thanks.

This message suggests that a previous post brought to the attention of this student
the idea of using peers at another location to review her test. However, the idea was
new to the student and needed to be tested out in the field. It is possible that some
other students in the class had relevant experience and could provide valuable feed-
back to this student. Unfortunately, this conversation was not followed up in the Fall
semester.

Comparing the Impact of Two Different Designs for Online Discussion 373
Such follow-ups, however, were common in the Spring section of the course. For
instance, one student responded to a previous post by suggesting a new idea for pre-
testing when validating a test:

I believe the pre-test is a good idea, but why does it need to be a test? An-
other good way to test if the students know the material could be by a game,
such as a relay game, or different things like that. This material you cover in
the game would be related to what the test is going to be like and then if the
students have [a] question they are more than welcome to ask… Adding in
some fun and action will allow the students to be more involved and feel
more comfortable on the material.

The message that immediately followed this post provided additional support for
an alternative pre-test, as well as reflecting upon other ideas for test validation:

I completely agree that the "pre-test" does not have to be an actual test. Mak-
ing it a test penalizes the students for not knowing information that they may
have never been exposed to, and [it is inappropriate] since you're mostly us-
ing it to assess what they do and don't know in order to better plan your
curriculum…

The first post introduced the idea of using a game as an alternative pre-test and
suggested that this approach would help with student motivation. The second post
added to this argument from a different perspective, by suggesting that providing an
alternative pre-test was not only useful but also very necessary. Such posts constituted
evidence that new learning had occurred among the students, as a result of the follow-
ups of messages made possible by the interactions of the students in the Spring
semester.
4 Conclusions and Discussions

This study examined the impact of two different designs of online discussion on stu-
dent participation in the discussions and the type of learning that was generated. The
results showed that when students were given an opportunity to respond to each other
with the instructor mostly absent in the discussion process until the end of the discus-
sions, they would visit the discussion forums more frequently. They were also more
likely to pose follow-up questions for their classmates and engage in meaningful
inquires. On the other hand, when the instructor posted frequently in the discussion
forums, students would respond to him instead of their peers, and the discussions
were seldom carried forth in much depth.
The findings confirmed what was reported in the research literature regarding the
negative relationship between instructor involvement and student participation in
online discussions [11]. Although the instructor may need to provide more detailed
feedback at the end of the discussion, the release from making frequent visits and
posts in the forum would save the instructor valuable time. When online discussion is