Purdue University

Purdue e-Pubs

Purdue Writing Lab/Purdue OWL Research

Reports

Purdue Writing Lab/Purdue OWL

2006

Purdue Online Writing Lab (OWL) Research

Report

Michael Salvo

Purdue University

Follow this and additional works at: h8p://docs.lib.purdue.edu/writinglabreps

Part of the Art and Design Commons, Cataloging and Metadata Commons, Higher Education

Commons, Instructional Media Design Commons, and the Rhetoric and Composition Commons

7is document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact epubs@purdue.edu for

additional information.

Recommended Citation

Tammy Conard-Salvo, Associate Director, Purdue Writing Lab

Jo Doran, Ph.D. Student in Rhetoric and Composition, Purdue University

Morgan Sousa, Ph.D. Student in Rhetoric and Composition, Purdue University

1

2

3

However:

• Participants wanted design features the redesigned OWL does not currently offer

• Participants took time and number of mouse clicks to complete some tasks than expected

• Participants could not complete some tasks

• Some participants’ responses to the redesigned OWL were neutral, which does not

represent the positive impression the Writing Lab desires for its new OWL.

second, previous and new user impressions of the OWL are different. Participants who visited

3. Add a search function

4. Incorporate graphical logos in the OWL Family of Sites homepage

5. Continue testing to measure usability and to generate new ideas for design and content.

Online Writing Lab programmers have integrated some of these changes, and overall, we believe

the redesign is a success. Test participants call the new site “impressive” and “a great site.”

work, but it’s better than the old site!” Theory-based, data-driven updates on the redesign

continue, and usability testing will work to help the Writing Lab and its OWL users. We believe

that the information presented in this report, as well as other open-source venues connected with

4

July 2006. The report presents our recommendations for the OWL based on data-driven findings

and user-centered theory. Finally, the report discusses broader implications and applications of

1. To provide researchers and pedagogues interested in usability, user-centered theory, and

usability is a critical concern.

2. To provide Writing Lab researchers, administrators, and pedagogues information on

improving the usability of online resources.

employ designers, or usability experts. In addition, it is often beyond the scope and

how best to employ usability research methods.

That is, we have an interest in promoting publication of effective and usable web-based

to their local project goals, we hope this report will help them identify the methodologies and

5

• Professional communication: esp. usability, user-centered theory, participatory design

• Writing Labs and OWLs

and non-technical readers alike. Please contact the research group to clarify technical detail,

expect. The OWL redesign does not provide participants with as much navigational information

as it should, thereby leaving participants indifferent in their impressions of their experience with

the OWL. Although inconclusive, three areas of interest for future research follow: gender-based

usage patterns, second and foreign language acquisition issues, and first-time and returning user

1. Design links and pages for the types of visitors using the OWL (user-based taxonomy)

2. Move the navigation bar from the right side to the left side of the OWL pages

3. Add a search function

4. Incorporate graphical logos in the OWL Family of Sites homepage

5. Continue testing to measure usability and to generate new ideas for design and content.

6

resources from the original to the redesigned OWL, taxonomic challenges will continue to grow

describes the importance of a writing center being a welcoming space that is accessible and

usable for writers. A goal of usability testing, then, is to make physical space, virtual space,

product, and document as usable and accessible as possible.

Harris (1992) and Lunsford (1991) discuss the importance of writing center collaboration in

student writing processes. In the articulation of our usability testing methodology and methods,

we viewed our OWL as a constantly evolving, complex web of virtual texts. One of our core

goals of testing was to elicit feedback from participants regarding the OWL’s strengths and areas

for improvement. The collaborative nature of our usability testing can be likened to a tutorial

where real users interact with our site, provide their preferences, and collaborate on revision.

The following section outlines the background of the Purdue Writing Lab and the OWL, and

its history of excellent in-person, one-on-one tutor-based writing instruction. By providing

access to innovative writing resources, the OWL is part of this history because it promotes global

guidelines, providing improved access and fostering resource sustainability.

7

April 30, 2006:

To help the OWL better serve its users, technical coordinators use precise data collecting

applications to track OWL users while they navigate the site. Based on needs highlighted by the

• Maintain writing support for all users

• Develop library-like features

• Achieve a more interactive environment

• Ensure accessibility of materials and navigability of the site, including 508 compliance

• Transition from a print-based to an electronic-based culture (the Web)

8

4. The resource pages and specific handouts participants were asked to find.

We did not construct the test to measure and track the usability of the entire OWL Family of

users worldwide. However, we believe that we tested a diverse enough range of participants to

this theory, we asked a number of computer-related questions in our demographic survey

(reference demographic information below).

2. Site usability test

3. OWL feedback survey.

Testing materials are attached in Appendix 1. Tasks are described in the sections below.

This test is referred to as Task 1a., though the sequence of tasks was randomized throughout

prototypes, each of which differed to accentuate a particular type of layout change in potential

redesign. Laying out the designs in sequence allowed participants to consider a type of major

15

Testing was conducted by showing participants four prototypes with different visual navigation

As participants selected a prototype, similar prototypes were displayed. In each grouping, each

icons arranged vertically across the page, prototype 5 presents these icons in larger format

Each grouping of prototypes offers similarly structured redesign options. Each task asked

participants to assess the professionalism and apparent ease of navigation of the prototype, and

each participant first selected among prototypes 1, 4, 7, and 10. When the participant-selected

prototype group (A, B, C, or D) was displayed, the participant was again asked to rate the

16

At this stage of testing, all the remaining prototypes—2, 3, 5, 6, 8, and 9—are displayed for the

test participant and asked after one minute if there are any designs that should replace his/her

selection for the new OWL website design. This participant says no, that prototype 10 should be

As described above, each testing group had specific visual elements changed in order to test

navigation column. Prototype 3 offers a streamlined search bar, as well as News, Feature and

Contact links in a right-hand search bar.

links presented in an upper left-hand column, a search bar in the upper right-hand column, and

17

This task was designed to allow a maximum level of participant input and control over the design

of OWL homepage paper prototypes. In evaluating Task 1b, coders have established a naming

design and most frequent placement of these items on the screen.

revealing a limit to the effectiveness of open-ended questions for meaningful statistical results.

While the results may be more difficult to analyze leading to difficulties in drawing conclusions

for researchers, this is nevertheless a valuable testing regimen because it built rapport with test

through reliance on the more statistically rigorous but less dialogically oriented testing. In other

words, Task 1b accomplished its goals: increasing comfort of test participants and creating

opportunities for discussion with participants.

experiences. Part one of Task 2 asked participants to use the OWL to answer a series of four

clicks to new locations were recorded for each participant. Part two of the task asked participants

Our rationale for using this task is multi-layered. On the surface, we wanted a test of the actual

site included in our testing procedure. We also wanted to collect quantitative data about the site

wanted to compare our data from Tasks 1a and 1b to what we found in Task 2, both with the

participant preferences as well as their feedback on the functionality of the test.

18

By collecting quantitative and qualitative data, we were able to further triangulate our results to

present a more comprehensive picture of how participants feel about design and functionality.

After Tasks 1a, 1b, and Task 2 were completed, participants were asked to fill out a feedback

survey rating their experiences. A summated rating scale (Likert scale) was used to rate

participants’ responses on a number of areas, including how they felt while using the site, and the

quantitative results.

19

category/professional status.

2

20

they took first-year composition at Purdue or elsewhere, and we asked them about their

perceived writing fluency. Sixteen of our 18 participants reported taking first-year composition;

triangulate with those in the second part of our demographic survey. Participants reported a mean

in writing. These two questions highly correlated

reported a mean of 3.167 out of 5 for the statement “I do not enjoy talking about my writing” and

preferences, and feelings about technology and writing. Our participants reported high levels of

comfortable with building and maintaining websites (Mean 2.556) and only marginally

3

4

21

of writing is covered in more detail in our discussion section. When asked where participants

The G1 usability test showed that participants liked and found a number of elements useful in the

redesigned OWL. Participants were fond of the new design of the OWL Family of Sites

homepage. However, all three task areas of the test reveal disturbing trends that may undermine

the goals of the redesigned OWL. For example, while some tasks took participants a short time

to complete, one minute or less, some tasks took participants up to two minutes or more to

complete. Ten tasks took five-plus minutes to complete, and four participants could not finish

their tasks. While some tasks required three clicks or fewer to complete (the industry standard),

confused while using the OWL

• Participants who used the OWL before usability testing rated the site significantly higher

responded more positively.

When asked about what features could be improved or included in the OWL, participants noted

that moving the navigation bar to the left side of pages and adding a search function would help.

Participants preferred to see more helpful information on the homepage such as links for

different types of visitors (teachers, students, ESL learners) and most popular resources. Also,

22

following presents the findings as collected during testing.

Out of our 63 total suggestions, 20 (or 31.75%) involved a search bar (see Appendix 2 for a

breakdown of responses). Although our paper prototype does not present conclusive findings

participants suggested that we use left-hand navigation in various pages, while five other “like”

Prototype 1 was chosen a total of 7 times out of 47 recorded choices, or 14.89%. Number 1 was

23

Many of the preferences participants reported were answers to our prompts, answers relating to

ease of use, professional, writing-related links. However, a number of likes, dislikes, and

suggestions occurred outside of our prompts throughout the testing. These included preferences

about search bar existence and location, and the amount and organization of page content. We

also had many participants comment on the inclusion of specific content areas: resources for

various users, most popular links, a search function, contact information, and a link to frequently

asked questions. Participants were more likely to want these additional resources on their pages

and based many of their choices on the absence or presence of these resources. As prototype 10

24

2. Overall, males had a much larger range of answers to questions (for example, the finding

question about site navigation had a female range from 3-5 and a male range of 1-5).

3. We found a significant difference (p<0.05) in the scores relating to the information

accessibility question (prototype 3) and a score approaching significance on finding

4. Female participants reported that information was much more accessible than male

participants and evaluated the site as more usable than males. However, females

evaluated their response to the experience using the site lower than male participants.

However, after further testing, these results were not as clearly indicative of gender-based

differences in web surfing styles. See the Conclusion below (esp. page 32) for more about the

combined G1 and G2 data relating to gender.

When we combine the participant responses for G1 and G2 (the testing conditions were similar,

again found a strong significant difference (p<0.01) for the question “How did you feel when

specific information and the organization of the homepage (with previous users rating the site

31

The list below represents some findings based on our participant responses to the feedback

• Of our 18 participants, 11 (61.1%) indicated that they had visited the Purdue OWL.

• Seven (63.6%) of the 11 participants indicated that they have visited both the old and new

OWL sites.

• Of those 11 participants who had visited the OWL, 8 or 72.7% indicated that they only

visit it yearly; the other three indicated that they visit it monthly.

experiences above average (mean of 4.14) while new users rated their experiences as

neutral (mean of 3).

• Furthermore, we found significant differences (p<0.05 level) for our user experiences in

their perceptions of the organization of the home page (new users rated it at a mean of

• 72.7% of our previous users only visit the site once a year and yet their experiences are

question of how frequency of use impacts site usability.

32

used our site. User histories, in other words, are incredibly important to usability.

either neutral or somewhat positive about the OWL website design. Through further refinement

and clarity in taxonomy, organization, and navigation, the research group aims to improve these

initial impressions to positive responses rather than user indifference.

OWL users, we did find highly significant differences in how participants rated their experiences

New OWL Use

33

However, the test also revealed that some participants required one minute or more to complete

complete, and four tasks could not be completed at all. The shortest period necessary for

complete a task was ten minutes.

The site usability test showed that many tasks required three or fewer mouse clicks to complete.

Three clicks to destination is currently considered the commercial standard for navigation,

test also revealed that an alarming number of tasks required four or more clicks to accomplish

number of neutral responses that indicate the necessity for more work on usability. Based on the

goals of the redesigned OWL—positive feedback from participants—the number of neutral

responses on the feedback survey justified a second generation of testing (see below).

The site usability test suggested that males and females possess very different searching styles.

Specifically, at the start of the site usability test, females required much more time but many

fewer mouse clicks than males to locate pieces of information. Males, on the other hand,

section details the G2 usability test.

34

G2. We find that the OWL redesign improves on the original OWL, but that usability problems

still exist and should be addressed if the new OWL is to meet Writing Lab goals. We also find

• Move the OWL navigation bar from the right side to the left side of the OWL pages,

• Add an OWL search function,

• Incorporate logos in the OWL Family of Sites homepage, and

• Continue usability testing to measure success and stimulate new ideas.

After G1 testing, the OWL Usability Group concluded that more data was needed to provide

minor adjustments to methods and setting would have to be made in order to collect additional

data. Therefore, to collect more data on OWL usability, and to help refine our methods, we

proposed the G2 usability test. Second-generation testing allowed us to expand our knowledge of

accurate and usable information possible.

The test methods for G2 mirrored the test methods in G1 with two exceptions: testing the user-

method and tasks used in G1:

• The demographic survey (always proceeded the following tasks)

• The paper prototype activity (administered in two parts: 1a, ‘choose a paper prototype’

and 1b, ‘create a paper prototype’)

35

As in G1, we randomly altered the order of the tasks to decrease the chance that participants

participants to find information on the OWL prototype, such as punctuation handouts for K-12

information participants found and used on the OWL prototype, and we measured the amount of

time and number of mouse clicks it took to access this information. Finally, we asked

Two key differences distinguish G1 from G2 testing:

1. The first key difference between the G1 site usability test and the G2 test is that

student, or a student from a university other than Purdue) to match the categories of users

36

Test leaders followed a script but often added questions based on participants’ responses and

actions during the tasks. Test leaders explained to participants that test recorders could not

answer any questions and that interaction must only occur between test leaders and participants.

purposes of data collection, participants’ information was identified only by their test number.

addition, G2 was not constructed to measure and track the behavior of every type of user visiting

the OWL. By soliciting for participants on the Purdue campus in the summer months, we knew

we would assemble primarily upper level undergraduate or graduate students and staff from

Purdue rather than the full range of OWL users.

37

Since one purpose of our demographic data collection was to measure total computer knowledge

with several overlapping questions, we ran correlations between questions relating to Web

design and expert computer usage. A correlation between G2 participants of Web design and

expert status reveals only a 58% correlation (compared to G1 testing, which had a 72%

between G1 and G2 participants.

handedness did not have a bearing on the results of G2 tests, which mirrored results from G1,

where handedness did not have a bearing on results.

Participants were asked to report their knowledge and use of the OWL. Half (7 or 50%) reported

that they use OWL yearly. Four participants (28.57%) reported that they use OWL monthly

while 2 (14.28%) have never used OWL. One (1) participant (7.14%) did not respond.

Comparing this to G1 data, we have a decrease in the number of participants who have never

38

Our findings for the paper prototype tests mirror results from G1. These include the following:

• All (100%) G2 participants want a search bar included in the site. In the create-a-paper-

prototype task, all fourteen participants included a search bar in their site. Many

participants preferred pages that had search bars over those that did not.

• All (100%) G2 participants want a “most popular resources” box on the main page. In the

• Participants were also interested in seeing a Frequently Asked Questions box on the main

In addition to the general findings, several interesting individual preferences arose during testing:

1) participants as visual learners; 2) audience-based information (gender, ESL).

One female participant, who indicated that she was a visual learner throughout the test,

below is a picture of this participant’s Task 1b test result.

39

_____

We discuss more on multiple navigation taxonomies in the “OWL Prototype Findings” section

40

We concur with most of the suggestions in the preliminary report and have included some

original OWL, usability problems still exist and should be addressed if the new OWL is to meet

during testing indicated that usability testing can work as a user-centered method of invention to

help designers work on prototypes. Previous and new user impressions of the OWL are markedly

for the new design. Overall, this report concludes that the OWL redesign is a success.

support the recommendations made after the first generation of testing. The preliminary results

47

When comparing the Likert after-test questionnaire, we find two questions that are approaching a

more time clicking and searching through the site than their native-speaking counterparts, they

It is unclear why our non-native speakers rated the site higher than our native speakers, although

searching styles. Specifically, females, at the beginning of the test, required more time and fewer

mouse clicks to locate pieces of information within the OWL. Males, on the other hand, needed

question we asked. Questions about evaluating the site were ranked equally or higher by female

48

that females typically rate their computer “self-efficacy” lower than males.

control of the Internet. Further, research in the areas of gender and communication technology by

New findings surface when combining the G1 participants with G2 participants:

49

50

Interaction with participants and findings can lead to new questions and research ideas while

avoiding the ethical pitfalls that are a major concern in our discipline. For example, in their

Gesa Kirsch (1996) ask the following questions of research: “Who benefits from the research?

in the form of more usable services and products for research participants.

writing experts to design and administer the test and to analyze the results. Collaboration

between professional writing faculty, writing center administrators, graduate students, and

changes and continue to address the needs of users, lab staff, faculty, as well as undergraduate

and graduate researchers. We hope that this project and its findings will form a framework of

continuing research on Internet-based usability that will ensure premium services for the large

51

Blythe, Stuart. “Wiring a Usable Center: Usability Research and Writing Center Practice.”

Cassell, Justine. “Genderizing Human-Computer Interaction.” The Human-Computer Interaction

Ehn, Pelle. “Scandinavian design: On Participation and Skill.” Usability: Turning Technologies

Harris, Muriel. “Collaboration Is Not Collaboration Is Not Collaboration: Writing Center

(1992): 369-83.

<http://writingcenters.org/>.Jeffrey, Rubin. Handbook of Usability Testing: How to Plan,

Kirsch, G and Mortensen, P., eds (1996). Ethics and Representation in Qualitative Studies of

Latour, Bruno. Aramis or The Love of Technology. Trans. Catherine Porter. Cambridge: Harvard

52

United States Congress. Library of Congress. Library of Congress Website. Feb. 6 , 2006. Feb. 7,

2006 <http://www.loc.gov/index.html>.

Associates, Publishers, 2003. 441-463.

North, Stephen. “The Idea of a Writing Center.” College English 46 (1984): 433-446.

Lafyaette, IN. 7 February 2006 http://owl.english.purdue.edu/.

Sanchez-Franco, Manuel J. “Exploring the influence of gender on the web usage via partial least

squares.” Behaviour & Information technology 25.1 (Jan.-Feb. 2006): 19-36.

Tsai, Chin-Chung and Chia-Ching Lin. “Taiwanese Adolescents’ Perceptions and Attitudes

2004): 725-734.

53

set of strategies and principles that propose a theoretically informed set of guidelines for

the implementation of networked computer technologies in writing centers in particular,

and for writing instruction in general. The dissertation includes a study of Purdue’s

Blythe, Stuart. “Wiring a Usable Center: Usability Research and Writing Center Practice.”

21-31.

The article presents the results of a qualitative study examining gender differences in

Information and Communication Technologies (ICT) anxiety levels. Results of the study

show that females had more negative attitudes toward computers and the Internet than

Further, while computer experience decreased computer anxiety for men, the same effect

was not found in women.

Cassell, Justine. “Genderizing Human-Computer Interaction.” The Human-Computer Interaction

54

Inc, 1996.

The author, an instructor of technical writing, technical editing, human factors and

book includes the psychology and physiology of how users access, learn, and remember

Dumas, Joseph S., and Janice C. Redish. A Practical Guide to Usability Testing. Portland:

for assessing and assuring usability; they outline steps for planning and conducting a

usability test; and they explain data analysis and how testing can improve products and

processes.

55

He begins with a historical overview of the problem of technological use, and he

elaborates on user-centered theory by defining three focal issues of the theory: user

aimed at training readers in the field of technology studies and at experimenting in new

Marcus, Aaron. “Global and Intercultural User-Interface Design.” The Human-Computer

56