Presentations

Listening to Student Voices: Assessing and Responding to Students' Experiences with Blackboard Learn

Research Report

Dr. Suzanne L. Pieper
Coordinator for Assessment, e-Learning Center, Northern Arizona University

Linda Neff
Instructional Technologist, e-Learning Center, Northern Arizona University

July 1, 2011

Small image of larger PDF poster summarizing research resultsNote: This report is also available as a PDF, and the results are summarized on a PDF poster.

Contents

Introduction

This report describes a study of student experiences with Blackboard Learn (Bb Learn) that was conducted at Northern Arizona University (NAU) during the spring of 2011. The report begins with a review of the literature on student experiences with LMS and the value of this study to NAU. Next, the research questions guiding this study are listed, followed by a description of the survey instrument used for the study, the study participants, and the data analysis procedure. Finally, the results are presented along with a discussion and recommendations for course design.

Review of the Literature

Learning management systems (LMS), sometimes called course management systems, have become almost ubiquitous in higher education over the past ten years. According to a recent study of nearly 40,000 undergraduate students at 100 four-year institutions and 27 two-year institutions, 9 out of 10 student respondents indicated that they have used an LMS (Smith & Caruso, 2010). Additionally, "more than 90% of institutional respondents confirmed that they have at least one commercial, homegrown, or open source" learning management system (Smith & Caruso, 2010, p. 80).

What is a learning management system? Definitions vary a bit, but basically it is a software system designed to "manage learning-related materials and student-learning processes" (Chou, Peng, & Chang, 2010, p. 1004). Students and instructors use LMS not only in online courses, but in hybrid and web- enhanced courses, too. With the rapid growth of LMS over the last decade or so, educational researchers and practitioners are scrambling to study the effect of these systems on teaching and learning. However, much of the research to date has focused on faculty use and perceptions of the LMS. Often, student voices concerning their experiences with the LMS go unheard in the literature (Yuen, Fox, Sun, and Deng, 2009), with the exception of a few current studies.

In a recent study, Ioannou and Hannafin (2008) examined student attitudes about the efficiency, ease of use, and perceived usefulness of features that could potentially be added to an LMS. The researchers found that while most students were generally satisfied with the institution's LMS, they would have liked a more personalized learning environment. Similarly, Chou, Peng, and Chang (2010) found that students wanted more from their LMS. Specifically, students required more content-related interactive functions than were currently available in the university LMS. Yuen, Fox, Sun and Deng (2009) examined student experiences with LMS across colleges at a comprehensive university in Hong Kong and found significant differences between academic levels and gender in students' use and perception of the LMS. More undergraduates used instant messaging, blogs, and wikis than graduates, while more graduates used RSS and social bookmarking than undergraduates. Additionally, undergraduates perceived wikis more useful in their studies than graduates, while graduates perceived email, blogs, RSS, social bookmarking, and voice over IP as more useful in their studies than did undergraduates. The researchers also found that more male than female students used wikis. Pretorius and van Biljon (2010) found other differences in how students experience an LMS. Using data from eye tracking studies, questionnaires, and interviews, the researchers studied students who performed tasks on the university LMS. Differences were found in usability of the LMS depending on the internet experience level of the students.

Building on the research to date on student experiences with an LMS, the current study will address questions concerning ease of use, useful features, assistance with student learning, and overall impressions of Northern Arizona University's recently-adopted learning management system, Blackboard Learn. Differences in student experiences will also be examined by class standing, gender, and technology skills.

Value of Study to Northern Arizona University (NAU)

The primary goal of this study is to inform and improve the teaching, learning, and design of future courses to be taught in Blackboard Learn at NAU. Results of the study and recommendations for course design will be available for use by all faculty and support staff involved in designing online, hybrid, or web-enhanced courses in Blackboard Learn. At NAU, course design is typically a collaborative effort involving course instructors, instructional technologists, instructional designers, assessment specialists, media specialists, and librarians. While we believe that the LMS should not drive course pedagogy and practice, we also believe that understanding what worked and what did not work for students in Blackboard Learn is important for designing the best possible learning experiences for students.

Research Questions

This study will answer the following research questions:

  1. How easy was it for students to use Blackboard Learn?
  2. Which features of Blackboard Learn were most useful to students?
  3. How did Blackboard Learn assist with student learning?
  4. What were students' overall impressions of Blackboard Learn?

Instrument, Participants, and Procedure

The data for this study were collected from the Blackboard Learn Student Survey, a 57-item survey initially administered to gather feedback from students for the purpose of making a decision about which learning management system (LMS) would be adopted by Northern Arizona University. The survey consisted of Likert-scale items as well as open-ended responses. The survey was constructed as a form in Google Docs and was administered online to 1,005 students enrolled in 14 online courses offered in Blackboard Learn and 17 courses offered in Moodle during the summer and fall of 2010. Of the 1,005 students, 255 gave their consent to participate in the survey, as required by the university's Institutional Review Board. One hundred twenty-six students completed the survey in Blackboard Learn courses, and one hundred twenty-nine students completed the survey in Moodle courses.

For the current study, the survey data were mined for additional insights about students' experiences with Blackboard Learn. First, student responses on Likert-scale items that answer the research questions were selected for statistical analysis. Descriptive statistics were reported, and an inferential technique was used to analyze and report on differences in student perceptions of Blackboard Learn by class standing, gender, and technology skills. Second, students' open-ended responses were analyzed using Glaser's Grounded Theory method of comparative analysis. Responses were coded and categorized as themes emerged from the data. This approach "provides an opportunity for *student+ voices to be acknowledged without constraint or bias by the researcher" (Treml, 2010). Two readers were used to code open-ended responses in order to increase the reliability of the analysis.

Results

The Data Sets

For analyses of research questions 1, 3, and 4, a dataset of 124 cases was used. One hundred twenty-six cases were included in the original data file imported from Google, and two cases were deleted. One case was deleted because the respondent was a faculty member. Another case was deleted because the student took the survey twice in the same class, once earlier and once later. Only the later response was kept. One student took the survey twice—once each in two different classes—and both responses were kept.

For analysis of research question 2, results of the Likert-scale item analyses were reported only for students who used the feature. Consequently, sample sizes varied.

For the analyses of the demographic characteristics of the students, a dataset of 123 cases was used. Only one of the responses from the student who took the survey twice in two different classes was kept. When inferential techniques were used, academic levels of freshman, sophomore, junior, and senior were coded as 1. Graduate, non-degree graduate, and post-baccalaureate undergraduate were coded as 2. For level of technology skills, beginner and intermediate were coded as 1. Advanced was coded as 2.

Demographics

In order to assess the representativeness of the survey sample compared to the student population of NAU as a whole, the gender, academic level, ethnicity, and age groups of students who responded to the survey in the summer and fall of 2010 was compared to all students enrolled at NAU during the fall of 2010. The results of the demographic analysis, shown in Table 1 , indicate that the survey sample was reasonably representative of NAU students as a whole.

Table 1: Demographics of Survey Sample Compared to All NAU Students
  Survey Sample N=123 All NAU Students N= 25,204
Gender
Male 39 (31.7 %) 9,906 (39.3%)
Female 84 (68.3 %) 15,298 (60.6%)
Academic Level
High School Current   57 (.2%)
Freshman 4 (3.3%) 5,681 (22.5%)
Sophomore 38 (30.9%) 3,704 (14.6%)
Junior 25 (19.5%) 4,848 (19.2%)
Senior 27 (22%) 5,267 (20.8%)
Non-degree Undergraduate   214 (.8%)
Post-baccalaureate Undergraduate 2 (1.6%) 423 (1.6%)
Graduate 27 (22%) 4,744 (18.8%)
Non-degree Graduate 1 (.8%) 266 (1%)
Ethnicity
American Indian/Alaska Native 6 (4.9%) 1,214 (4.8%)
Asian 2 (1.6%) 468 (1.8%)
Black/ African American 7 (5.7%) 823 (3.2%)
Hispanic/Latino 17 (13.8%) 3,843 (15.2%)
International 2 (1.6%) 866 (3.4%)
Native Hawaiian/Other Pacific Island   109 (.4%)
Not Specified 2 (1.6%) 320 (1.2%)
Two or More 2 (1.6%) 531 (2.1%)
White 85 (69.1%) 17,030 (67.5%)
Age Group
Under 21 39 (31.7%) 9,653 (38.2%)
21-29 48 (39%) 9,841 (39%)
30-59 36 (29.3%) 5,602 (22.2%)
60 or Over 60   108 (.4%)

Level of Technology Skills

Six students (4.8%) rated their level of technology skills as beginner, 81 students (65.3%) rated their level of technology skills as intermediate, and 37 students (29.8%) rated their level of technology skills as advanced.

Research Question 1: How easy was it for students to use Blackboard Learn?

One item from the survey was selected to answer this research question: "Blackboard Learn is easy to use." Students answered this item using a four-point Likert scale: strongly agree, agree, disagree, and strongly disagree.

Ninety-three students (75%) indicated that they strongly agreed or agreed that Blackboard Learn was easy to use. The number of student responses on each point of the Likert scale is shown in the chart below:

Pie chart showing student responses to Blackboard Learn is Easy to Use. 26.6% strongly agreed. 48.4% agreed. 17.7% disagreed. 7.3% strongly disagreed.

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the item "Blackboard Learn is easy to use."

The t test comparing the mean scores of undergraduates and graduates on the item "Blackboard Learn is easy to use." found a significant difference between the means of the two groups. The mean for undergraduates was significantly higher than the mean for graduates. In other words, undergraduate students found Blackboard Learn easier to use than graduate students. Descriptive statistics and the results of the t tests conducted to examine group differences on the item "Blackboard Learn is easy to use." by gender, class standing, and technology skills are displayed in Table 2.

Table 2: Ease of Use of Blackboard Learn by Student Groups
Groups Mean SD N t
Gender
Male 2.92 .92 39 -.17
Female 2.95 .82 85  
Class Standing
Undergraduate 3.05 .88 94 2.57*
Graduate 2.60 .67 30  
Technology Skills
Lower Level 2.93 .85 87 -.24
Upper Level 2.97 .86 37  

Note: * = p < .05

Students were also asked to comment on the ease of use of Blackboard Learn, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments generally blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 3 summarizes the coding of 60 student open-ended responses:

Table 3: Codings and Number of Comments for Ease of Use of Blackboard Learn
Code # of Responses
Positive 8
Negative 29
Mixed 22

The chart below shows the positive themes regarding ease of use of Blackboard Learn and number of student comments for each theme:

Bar graph showing number of student  comments for each of three categories of positive themes. 2 comments were about ease of use. 4 comments were about ease of navigation. 6 comments were about other positive categories.

The chart below shows negative themes regarding ease of use of Blackboard Learn and the number of student comments for each theme:

Bar graph showing number of student comments for each of 9 negative themes. 10 comments were about lacking notification. 10 comments were about navigation issues. 10 comments were about organization problems. 5 comments were about difficulty in learning. 4 comments were about lack of user friendliness. 4 comments were about other negative categories. 3 comments said students didn't like the system. 2 comments were about too many windows.

Most of the student responses to the ease of use of Blackboard Learn were either negative or mixed. The largest number of negative comments concerned the lack of notifications, navigation issues, and organization problems in Blackboard Learn. Students were frustrated when they did not receive notifications when something new was waiting for them in Blackboard Learn: an email from their instructor, a new grade, or a discussion post from a classmate. They also had difficulties with navigating the LMS. As one student commented,

"Blackboard Learn was not intuitive. It took so much time to navigate through and around it, I was wishing that I had Vista back! It's not the change that bothered me, just Blackboard Learn in itself."

Other students complained that the system seemed disorganized and that it took some time to find their way around the course. One student summed up the many frustrations with using Blackboard Learn:

"Why don't you tie a string to a can in Flagstaff and I can have the other end tied to a can here in Tucson, and maybe, just maybe, Blackboard Learn will come close to that."

Interestingly, while many students commented on navigation issues with Blackboard Learn, ease of navigation was also one of the most frequently cited positive aspects of the LMS. As one student commented,

"Bb Learn "looks" much different than Vista. Once I was able to get past the new look of Bb Learn, the system was easier to navigate. I like the Home page view because I can access grades, schedules, etc. easily."

Other students commented that Blackboard Learn is easy to learn and use, and one student especially liked the availability of video.

Research Question 2: Which features of Blackboard Learn were most useful to students?

A series of 11 items was selected to answer this research question. Students were asked to "Please rate the following feature in terms of how useful it is to you." and presented with 11 selected features of Blackboard Learn for their response. Students answered this series of items using a four-point Likert scale: strongly agree, agree, disagree, and strongly disagree with a fifth option of "I did not use this feature."

Students found the two features related to grades and instructor feedback most useful: getting assignments back with grades and comments and keeping track of grades on assignments and quizzes. These features were closely followed by listening or viewing media and accessing the online syllabus. As expected, accessing the online syllabus and keeping track of grades on assignments and quizzes were also the most used features. Table 4 displays the item means for each feature on a scale of 1 (Strongly Disagree) to 4 (Strongly Agree) and lists the features in order of most to least useful:

Table 4: Usefulness of Features in Blackboard Learn
Feature Mean SD N
Getting assignments back with comments and grades 3.48 .72 112
Keeping track of grades on assignments and quizzes 3.42 .72 123
Listening to or viewing instructional media 3.41 .70 103
Accessing an online syllabus 3.40 .63 124
Turning in assignments online 3.36 .75 116
Taking exams and quizzes online 3.31 .73 111
Accessing online readings and links to text-based materials 3.31 .74 121
Communicating with course email 3.18 .77 103
Sharing materials with other students 3.02 .80 100
Communicating using online discussion board 2.98 .90 98
Communicating with online chat 2.71 .94 71

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the 11 items beginning with the phrase "Please rate the following feature in terms of how useful it is to you." followed by 11 selected features of Blackboard Learn.

The t test comparing the mean scores of males and females on the item "Sharing materials with other students is useful to me." found a significant difference between the means of the two groups. The mean for females was significantly higher than the mean for males. In other words, female students found sharing materials with other students using Blackboard Learn more useful than male students. Descriptive statistics and the results of the t tests conducted to examine group differences on the item "Sharing materials with other students is useful to me." by gender, class standing, and technology skills are displayed in Table 5:

Table 5: Usefulness of Sharing Materials With Other Students by Student Groups
Groups Mean SD N t
Gender  
Male 2.75 .79 28 -2.13*
Female 3.12 .78 72  
Class Standing
Undergraduate 3.02 .81 74 .14
Graduate 3.00 .80 26  
Technology Skills
Lower Level 2.98 .82 73 -.68
Upper Level 3.11 .77 27  

Note: * = p < .05

No other significant differences were found when examining group means on the other items in this series.

Students were also asked to comment on the any features of Blackboard Learn that they liked or disliked, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments typically blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 6 summarizes the coding of 37 student open-ended responses:

Table 6: Codings and Number of Comments for Features of Blackboard Learn
Code # of Responses
Positive 4
Negative 18
Mixed 15

The chart below shows the positive themes regarding features of Blackboard Learn and number of student comments for each theme:

Bar graph showing number of comments about positive features of Blackboard Learn. 2 comments were about organization. 2 comments were about other features.

The chart below shows negative themes regarding features of Blackboard Learn and the number of student comments for each theme:

Bar graph showing number of comments about negative features of using Blackboard Learn. 5 comments were about the discussion tool. 4  comments were about email. 4 comments were about lack of notifications. 2 comments were about the gradebook. 2 comments were about assessments. 5 comments were about other features.

Most of the student responses to the features of Blackboard Learn were either negative or mixed. The largest number of negative comments was single complaints about features such as the HTML formatter, links to outside sources, and the wiki tool. An equal number of students complained about the discussion tool, including one student who had navigation issues:

"The discussion board was much different than any I had used before and was extremely hard to navigate."

Students also had a number of issues with lack of notifications and with email, including lack of accessibility. Students further commented on problems with the grade book and with quizzes and assessments. As one student commented:

"On the screen after an assessment is completed, the submit button is way on the bottom of the screen and while this is a minimal complaint, it is still annoying."

Very few students commented positively on Blackboard Learn features. However, students did comment on the organization and ease of use of Blackboard Learn as well as video accessibility and the ability to track grades.

Research Question 3: How did Blackboard Learn assist with student learning?

Two items from the survey were selected to answer this research question: "Using Blackboard Learn in this course has improved my learning." and "Blackboard Learn fits my learning style." Students answered these items using a four-point Likert scale: strongly agree, agree, disagree, and strongly disagree.

Seventy-two students (58.1%) indicated that they strongly agreed or agreed that using Blackboard Learn in the course improved their learning. The number of student responses on each point of the Likert scale is shown in the chart below:

Pie chart showing percentage of students who say Blackboard Learn has improved their learning. 11.3% strongly agreed. 46.8% agreed. 32.3% disagreed. 9.7% strongly disagreed.

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the item "Using Blackboard Learn in this course has improved my learning." No significant differences were found when examining group means on this item.

Students were also asked to comment on how Blackboard Learn assisted with learning, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments generally blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 7 summarizes the coding of 41 student open-ended responses:

Table 7: Codings and Number of Comments for How Blackboard Learn Assisted With Learning
Code # of Responses
Positive 11
Negative 15
Mixed 15

The chart below shows the positive themes regarding improved learning with Blackboard Learn and number of student comments for each theme:

Bar graph showing number of comments with positive themes about improved learning with Blackboard Learn. 5 comments were about organization. 2 comments each were about communication/collaboration, navigation, and other topics.

The chart below shows negative themes regarding improved learning with Blackboard Learn and the number of student comments for each theme:

Bar graph showing number of comments with negative themes about improved learning with Blackboard Learn. 9 comments said the system wasn't user friendly. 2 comments each were about quizzes, lack of improved learning, and other topics.

Student responses to improved learning with Blackboard Learn were equally negative and mixed. The largest number of negative comments concerned the lack of user friendliness of the system. As one student commented,

"I spent more time frustrated with the system than I did on the material itself."

Students also commented that technical issues, including problems with quizzes, impeded their learning. A couple of students concluded that Blackboard Learn did not help to improve their learning, as one commented,

"In no way, shape, or form has Bb Learn improved my learning. No way at all. As a matter of fact I have a good mind to see if I can get my money back..."

While many students complained that the Blackboard Learn system was not user friendly and actually impeded learning, other students liked the organization of the system. As one student observed:

"Once I learned how to use the system, I felt more organized than I usually do with other online courses."

Other students liked the communication and collaboration tools, including wikis, blogs, and discussion boards. As one student commented:

"In this course, we used the Wiki tool, blog, and discussion boards frequently ...our class was able to hold up conversations that really expanded my learning of the course subject."

Still other students liked the ease of navigation and the availability of video.

Ninety-one students (73.3%) indicated that they strongly agreed or agreed that Blackboard Learn fit their learning style. The number of student responses on each point of the Likert scale is shown in the chart below:

A pie chart showing the percentage of students who said Blackboard Learn fits their learning style. 18.5% strongly agreed. 54.8% agreed. 20.2% disagreed. 6.5% strongly disagreed.

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the item "Blackboard Learn fits my learning style." No significant differences were found when examining group means on this item.

Students were also asked to comment on how Blackboard Learn fit their learning style, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments generally blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 8 summarizes the coding of 27 student open-ended responses:

Table 8: Codings and Number of Comments for How Blackboard Learn Fit Learning Style
Code # of Responses
Positive 12
Negative 9
Mixed 6

The chart below shows the positive themes regarding how Blackboard Learn fit students' learning style and the number of student comments for each theme:

Bar graph showing positive comments about how well Blackboard Learn fits student learning styles. 4 comments each were about presentation of content and convenience/flexibility of online courses. 2 comments each were about learning well online and liking online learning tools

The chart below shows negative themes regarding how Blackboard Learn fit students' learning style and the number of student comments for each theme:

Bar graph showing number of negative comments about how well Blackboard Learn fits student learning styles. 3 comments were about not learning well online. 2 comments each were about system complexity, anxiety, and issues with the discussion tool.

Student responses to Blackboard Learn's fit with learning style were more positive and negative than mixed. The largest number of positive comments concerned the presentation of course material. As one student reflected:

"I prefer to learn visually. It's hard for me to stay motivated and focused when I have to click through a bunch of pages and obstacles to get to what I need. The openness of this system helps my learning by making course content readily accessible and easily viewed."

Other students commented that they liked taking online courses because they are convenient and they can learn at their own pace or that they just learn better online. As one student commented:

"I really liked being able to access the information on my own time and teach myself."

Students also commented on the benefits of having multiple learning tools available to them, particularly online discussions and virtual lectures.

Most of students' negative comments regarded not learning well in an online environment, in contrast to the students who enjoy and learn well online. One student commented:

"I have found that I cannot take an online only class. The best I could do would be to take a hybrid class as I still need some face to face interaction with the professor."

Other students noted that the system was too complex, that using Blackboard Learn created anxiety, and that discussions could be more accessible and easier to edit.

Research Question 4: What were students' overall impressions of Blackboard Learn?

Two items from the survey were selected to answer this research question: "I have had a positive overall experience using Blackboard Learn." and "I would like to take another Blackboard Learn course." Students answered these items using a four-point Likert scale: strongly agree, agree, disagree, and strongly disagree.

Ninety-one students (73.4%) indicated that they strongly agreed or agreed that they had a positive overall experience using Blackboard Learn. The number of student responses on each point of the Likert scale is shown in the chart below:

Pie chart showing the percentage of students who had a positive overall experience with Blackboard Learn. 22.6% strongly agreed. 50.8% agreed. 20.2% disagreed. 6.5% strongly disagreed.

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the item "I have had a positive overall experience using Blackboard Learn." No significant differences were found when examining group means on this item.

Students were also asked to comment on their overall experience with using Blackboard Learn, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments generally blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 9 summarizes the coding of 27 student open-ended responses:

Table 9: Codings and Number of Comments for Overall Experience with Blackboard Learn
Code # of Responses
Positive 5
Negative 10
Mixed 12

The chart below shows the positive themes regarding students' overall experience with using Blackboard Learn and the number of student comments for each theme:

Bar graph showing number of positive comments about Blackboard Learn overall. 3 comments said there were no major problems. 2 comments fell in other categories.

The chart below shows negative themes regarding students' overall experience with using Blackboard Learn and the number of student comments for each theme:

Bar graph showing number of negative comments about Blackboard Learn overall. 5 comments said it wasn't a good experience. 2 comments said the students preferred Vista. 3 comments fel in other categories.

Student responses to their overall experience with using Blackboard Learn were mostly mixed and negative. The largest number of negative comments regarded students' bad experiences with Blackboard Learn in general. One student was especially unhappy with the experience:

"I will avoid Bb Learn like the plague. If NAU adopts this platform I will seriously consider transferring to U of A just so I don't have to use a system like Bb Learn."

Other students commented on issues with navigation, quizzes, and email. Still other students simply preferred Vista over Blackboard Learn.

The majority of students who responded positively commented that they had no major problems with the system. One student who was a newcomer to online courses appreciated this aspect of Blackboard Learn:

"No major glitches to report, and that is all I ask for. For a first-timer, it was awesome. Thank you for your efforts to make that possible."

Other students commented on the format for discussions and the ease of access and navigation.

Eighty-six students (69.3%) indicated that they strongly agreed or agreed that they would like to take another course in Blackboard Learn. The number of student responses on each point of the Likert scale is shown in the chart below:

Pie chart showing percentage of students who would like to take another course in Blackboard Learn. 25.8% strongly agreed. 43.5% agreed. 20.2% disagreed. 10.5% strongly disagreed.

An independent-samples t test was conducted to examine the difference between the means of males and females, undergraduates and graduates, and lower-level and upper-level self-reported technology skills on the item "I would take another course in Blackboard Learn." No significant differences were found when examining group means on this item.

Students were also asked to comment about taking another Blackboard Learn course, and their comments were coded as overall positive, negative, and mixed. Next, themes were identified within these codings, with the exception of responses coded as mixed. Mixed comments generally blended positive and negative comments and were therefore difficult to analyze for themes. The theme of "other" was designated for single comments. Table 10 summarizes the coding of 26 student open-ended responses:

Table 10: Codings and Number of Comments for Taking Another Blackboard Learn Course
Code # of Responses
Positive 11
Negative 9
Mixed 6

The chart below shows the positive themes regarding students’ taking another Blackboard Learn course and the number of student comments for each theme:

Bar graph showing number of positive comments about taking another Blackboard Learn course. 4 each said students felt comfortable after learning the system or had no problems. 2 comments fell in other categories.

The chart below shows negative themes regarding students' taking another Blackboard Learn course and the number of student comments for each theme:

Bar graph showing number of positive comments about taking another Blackboard Learn course. 4 comments said no thanks. 2 comments said students preferred Vista. 3 comments fell in other categories.

Student responses to their overall experience with using Blackboard Learn were positive and negative. The largest number of positive comments regarded students' feeling comfortable with the system after having learned it or not having a problem with taking another course in Blackboard Learn. One student compared Blackboard Learn to Vista:

"I would definitely prefer Bb Learn over Vista in any future online courses. Bb Learn makes Vista look archaic! Bb Learn is simply more intuitive, and it just functions better for these courses than Vista ever has."

Other students liked the communication tools available and the ease of navigation and organization.

In contrast, students who responded negatively did not want to take another course using Blackboard Learn. As one student commented:

"I am glad I will be graduating before the transition from Vista."

Other students noted problems with the lack of notifications, the inordinate amount of time taken to learn the system, and the "awful experience" of taking a course in Blackboard Learn.

Discussion and Recommendations for Course Design

This study analyzed quantitative and qualitative data in order to answer questions about students' experiences using Blackboard Learn. While the Likert scale and t-test results provided an overall "feel" for student experiences with Blackboard Learn, the open-ended comments provided more specific information regarding what worked and what did not work for students. The following discussion considers what the data tells us about the research questions and provides some recommendations for how instructors might incorporate this information in the future design of Blackboard Learn courses.

Ease of Use

Although 75% of the students indicated on the scaled item that Blackboard Learn was easy to use, most students' open-ended comments concerning ease of use were either negative or mixed. The largest number of student complaints involved a lack of notifications, navigation issues, and organizational problems with Blackboard Learn. Graduate students found Blackboard Learn more difficult to use than undergraduate students. This finding makes sense given that graduate students are, for most part, older than undergraduate students and may not feel as comfortable learning via an LMS. On the other hand, all students liked the availability of video, the new Home Page view, and the fact that everything was organized and easy to learn.

After reviewing the student comments as a second reader for this study, Linda Neff, an instructional technologist with the e-Learning Center, made a number of recommendations for instructors using Blackboard Learn (2011). Overall, she suggested that instructors take advantage of the aspects of Blackboard Learn that students like, while minimizing or eliminating the aspects that created problems for students. For example, all instructors—undergraduate and graduate—can minimize the issue of lack of notifications by using the Home Page view as the entry page for students, always keeping the following modules in view: To Do, What's New, My Calendar, My Announcements, and Alerts. She also suggested that instructors consider incorporating online video in their courses, providing students with an alternative to text-based learning.

Useful Features

Students indicated on the scaled items that the most useful features in Blackboard Learn were those related to grades and instructor feedback, closely followed by listening to or viewing instructional media and accessing an online syllabus. The least useful features were communicating with online chat, communicating using the online discussion board, and sharing materials with other students.

Additionally, female students found sharing materials with other students using Blackboard Learn more useful than male students. The reasons for this finding are not particularly clear. Student open-ended comments reinforced the responses on the scaled items. For example, students had navigation issues with the discussion tool, but they liked the ability to track their grades in the LMS.

Neff recommended that instructors use the Discussion Board link in order to encourage student interaction, but incorporate small group discussions to reduce the number of discussion posts that students must read as well as to encourage deeper and more meaningful conversations. At the same time, she recommended that Blackboard redesign the Discussion Board link in Blackboard Learn to make it more user-friendly. Additionally, she recommended that instructors use the My Grades link in the course menu, post assignment feedback in a timely manner, and provide students with directions for viewing instructor feedback.

Assistance with Learning

Students responded to two survey items that answered this question: "Using Blackboard Learn in this course has improved my learning." and "Blackboard Learn fits my learning style."

Slightly over half of the students (58%) agreed on the scaled item that Blackboard Learn assisted with their learning. The largest number of student open-ended comments was either negative or mixed for this item. Most students complained that the lack of user friendliness of the system impeded their learning. In contrast, those students who responded positively commented that the organization of the system helped them to learn.

Instructors in Blackboard Learn can smooth the large learning curve for students by building in some time at the beginning of the course for students to view online e-Learning Center video tutorials, suggested Neff. Instructors can also build in some low-stakes activities at the beginning of the course for students to practice navigating the course, for example uploading assignments, taking a quiz, or reviewing their grades. Neff also recommended taking advantage of students' desire for organization by including a table of contents in each learning module to help students stay organized.

More students (73%) agreed on the scaled item that Blackboard Learn fit their learning style. Similarly, students' comments were more positive than negative or mixed. Once again, the largest number of students commented that they liked the organization of the LMS and the presentation of content. They also liked the convenience and flexibility of an online course in general. On the other hand, the greatest number of student complaints concerned the discovery that they did not learn well online.

Neff suggested that using a variety of instructional formats will optimize the learning of all students in an online course. She recommended, for example, creating online lectures that students can download and listen to or watch on their mobile devices, posting course readings online so that students can read at their own pace, and incorporating opportunities for interaction and collaboration using the discussion board, wikis, and blogs.

Overall Impressions

Students responded to two survey items that answered this question: "I have had a positive overall experience using Blackboard Learn." and "I would like to take another Blackboard Learn course."

Almost three-fourths of the students (73%) agreed on the scaled item that they had a positive experience using Blackboard Learn. However, students' open-ended responses to this item were mostly mixed and negative. Most students simply commented that taking the Blackboard Learn course was not a good experience for them. In contrast, students who had a positive experience basically commented that they did not have problems with Blackboard Learn.

Similarly, 69% of the students agreed on the scaled item that they would like to take another Blackboard Learn course. Student responses on the open-ended comments were more positive for this item. The largest number of students who would like to take another Blackboard Learn course responded that they felt comfortable with using the LMS after they learned the system. On the other hand, the largest number of students who responded that they would not like to take another Blackboard Learn course responded simply with "No thanks."

A common theme running through student comments for both of these items was comparing Blackboard Learn to Vista, either favorably or unfavorably. Blackboard Learn was a change over Vista, and some students adjusted to that transition better than others. Neff suggested some ways for instructors to make the transition easier for students. For example, in Vista students could easily see tools such as the discussion board when they joined their online course. However, in Blackboard Learn these tools are buried until students click on a Tools link. Neff suggested removing the Tools link altogether or giving it a name that makes sense to students. She also suggested simplifying the list of tools by keeping the tools that are being used and hiding the tools that are not being used in the course menu.

A Final Thought

The purpose of this study was to assess students' experience using Blackboard Learn and respond to what worked and what did not work for students with guidelines for course instructors and others designing courses in Blackboard Learn. As instructors use these guidelines in their courses, it will be important for them to keep in mind student voices, particularly the comment of one insightful student on the Blackboard Learn Student Survey. This student reminded us that the LMS is only a tool, and how instructors and students use the tool, not the tool itself, determines improvement in learning:

"The software management system didn't help me learn. It just provided the keys that I needed to improve my learning."

References

Chou, C., Peng, H., & Chang, C. (2010). The technical framework of interactive functions for course-management systems: Students' perceptions, uses, and evaluations. Computers and Education, 55, 1004-1017.

Ioannou, A., & Hannafin, R. (2008). Deficiencies of course management systems: Do students care? The Quarterly Review of Distance Education, 94 (4), 415-425.

Neff, L. (2011). Bb Learn Recommendations. Unpublished manuscript, e-Learning Center, Northern Arizona University, Flagstaff, Arizona.

Pretorius, M., & van Biljon, J. (2010). Learning management systems: ICT skills, usability, and learnability. Interactive Technology and Smart Education. 7 (1), 30-43.

Smith, S. D, & Caruso, J. B. ECAR study of undergraduate students and information technology (ID: ERS 1006). Retrieved from EDUCAUSE website: http://www.educause.edu/Resources/ECARStudyofUndergraduateStuden/217333.

Treml, M. (2010). Faculty attitudes and perceptions of the Liberal Studies Program: Report on a survey of faculty at Northern Arizona University. Unpublished manuscript, Office of Academic Assessment, Northern Arizona University, Flagstaff, Arizona.

Yuen, A., Fox, R., Sun, A. & Deng, L. (2009). Course management systems in higher education: Understanding student experiences. Interactive Technology and Smart Education. 6 (3), 189-205.