2.11.+Using+virtual+worlds+for+assessment+in+a+Languages+subject

=Using virtual worlds for assessment in a Languages subject: A case study=

This assignment comprised two parts and was conducted in two lessons during class, in a computer lab. In Second Life, students worked in virtual groups to complete a number of tasks, including ordering restaurant meals and buying ingredients from a market to make specific dishes. Students had to use their Chinese language skills to assess the options and make the correct choices. Assessment was based on correctly completing the specified tasks.

The assignment was worth 10% of total marks for the subject (Part A and B worth 5% each).

Web 2.0 tool used: Second Life @http://secondlife.com/

 **About the subject**
This was an undergraduate first year subject "Chinese 1", offered as part of a Bachelor of Arts degree. The subject was compulsory for some students and elective for the rest.

The subject provided an introduction to modern standard Chinese, both spoken and written, to students with no prior knowledge of Mandarin. The subject included an overview of the important sentence structures of modern standard Chinese with equal emphasis on the four skills of reading, writing, listening and speaking.

During this semester, approximately 140 students were enrolled in the subject. Approximately 70% of the students were local students, whose main language was English, and 30% were international students with a mixture of Asian and other language backgrounds.

The subject was taught by two lecturers and three Chinese national sessional tutors.

The subject has been taught for many years, but this Second Life assignment was only initiated in 2008. 2010 was the first time the Second Life assignment was formally assessed.

 **Assignment tasks and timing**
In Week 5 of the semester students took part in an introductory computer lab session, where they had to register for a Second Life account and then undertake a training course in basic Second Life skills.

In Week 8 students took part in a second computer lab class, in which they completed the first part of the assignment. During this class, students were asked to go to a tea house on the University’s Chinese Island in the virtual world of Second Life. They then had to form into groups of four and each member of the group was given a specific role to play with specific dietary needs (e.g. a vegetarian, a diabetic, a Muslim, a person who cannot eat spicy food). Each group then had to collect information about the ingredients of a range of dishes commonly found in mainland China (17 in all) and then discuss and decide on which dishes were appropriate for the specific dietary needs of each of four different roles specified as part of the task. Finally each group was required to order, in Chinese, the dishes they had decided on.

Teaching staff went around to each group and asked the group for their order in Chinese. Once the dishes were ordered, staff then recorded what dishes were ordered and checked that the dishes met the dietary requirements of the four dietary types (marks were then allocated on the number of appropriate dishes out of four required dishes).

In Week 12, students attended a third computer lab class. During this class they were asked to complete a series of steps to buy a bowl of soup dumplings for a hypothetical teacher. In order to do this they had to find and identify (through conversation in Chinese) a particular individual in the restaurant (tea house) on the University’s Chinese Island who had information needed to complete the assignment. Students then had to purchase a particular ingredient for the soup dumplings in the Chinese Island agricultural products market. They found the market by following directions provided by a non-player character (NPC) in the restaurant. They purchased the ingredient concerned by engaging in dialogue with another non-character player in the market (a fruit and vegetable seller). Having purchased the ingredient, students then had to return to the restaurant and, based on further instructions given to them by the NPC there, use the ingredient to ‘cook’ their own soup dumplings.

At the end of the allotted time, teaching staff then went around to each student to see what stage they had finished at that point in time. Marks were allocated on the basis of stage of completion. Each lab class lasted for 1.5 hours, and all tasks were to be completed during class time.

 **Intended learning outcomes**
The assignments were designed to develop students' Chinese communicative competence and cultural knowledge, using knowledge and skills derived from formal classroom learning; from the virtual environment of the tea house and the information contained therein; and through interacting with teaching staff, peer team members and the artefacts (including automated NPCs) in the virtual environment. The lecturer described the purpose of the assignment as follows:

"A key objective of the lessons was, in addition to providing meaningful communicative tasks/opportunities, to make students engage in authentic cognitive processes that reflect the processes that would occur in real life while dealing with the task/interlocutors in the target language. The deliberate inclusion of a number of ‘information gaps’ meant that students had to think their way through to the next move. One of the guidelines they were given for this was the question: what would you do in real life?"

 **Why Web 2.0?**
Second Life was chosen for this assignment to enable students to undertake the tasks in a setting that emulated a real-world context. This meant students were able to have an "authentic" learning experience, consolidating and building on previous knowledge.

The Chinese Island in Second Life was set up to provide a context for learning and practice that simulated real-life contexts. The environment contained many of the sensory and informational features of a similar real-world environment and interaction in the environment was "natural" in that it involved learning and practice while completing tasks that were relevant to the real world. The tasks involved the use of linguistic and cultural skills and knowledge in a purposeful way - not just "practising for the sake of practising".

Using Second Life also facilitated collaborative learning and peer scaffolding. Because the tasks were undertaken in a computer lab session, collaboration occurred both within the virtual environment and in the physical classroom.

A further rationale for using Second Life in this assignment was that it required students to type in Chinese characters to communicate with each other and with the non-player characters in the environment. To type in Chinese characters students needed a good grasp of the Romanised spelling of the characters (pinyin, which is a requirement of the formal curriculum and assessed separately). The language used in the typed exchanges was conversational rather than written, which had the potential to benefit students' oral conversational language skills. All text-based interaction could be automatically logged for post-lesson review.

In addition, other Web 2.0 learning environments could be easily incorporated into the Second Life lessons. Teaching materials, lesson plans, lesson objectives, links to inworld locations, quizzes, and signup sheets, were all provided to students through the University's Moodle learning management system.

 **Setting up the task**
The lecturer who coordinated the subject received a University teaching and learning grant, which partially enabled the construction of the environment. The coordinator designed the overall environment and negotiated specifics (including layout, content and price) with the contracted "builder" (who did the work from southern China). The coordinator developed additional virtual world materials, such as virtual artefacts (including non-player characters), soundscapes, and programming. However, these were not developed solely for the purposes of these particular lessons. They will be used facilitate a wide range of learning activities over a period of time.

The coordinator also developed all of the lesson plans, materials (incorporated into lesson manuals provided to students both electronically and in print) and specific lesson tasks. He located a suitable site within Second Life for students to undertake basic Second Life skills training and negotiated with the relevant organisation for the use of that site prior to the assessed lessons commencing.

 **Introducing the assignment to students**
When introducing the assignment, staff gave an explanation to students about the learning environment chosen, the reasons for choosing it and why lessons within the environment were a genuine learning opportunity.

In their first lab class, in Week 5 of the semester, students had to register a Second Life account, create an account on Moodle and then undertake a training course in basic Second Life skills that would facilitate subsequent lessons. This training took place in a region in Second Life called “Virtual Ability”, which has a basic skills orientation course set up for users of Second Life who are disabled in real life.

Students also watched two videos on inputting Chinese characters and were encouraged to do purpose-designed online exercises to practise typing skills (although less than half did this).

 **Supporting students through the assignment**
During the second assignment lab class, students were given a Lesson Manual that included refresher information about the basic technical skills relating to using Second Life, provided in written and pictorial format.

The content of the lessons and assessment was based on language previously learned in other parts of the curriculum (lectures, seminars, tutorials and the main textbook). Students were also encouraged to refer to their textbooks during the lab classes.

Lesson plans and objectives, specific task requirements and details on assessment were drawn up and distributed to all students via Moodle, and in in the form of Lesson Manuals. Other key information was also provided in text-based form within Second Life on notecards and in the content of conversations with the non-player characters.

The Lesson Manual also included additional background cultural information (textual and pictorial), new word lists and suggested expression lists (for conversation with the non-player characters in the second assessed lesson).

Prior to students entering Second Life for the second assessed lesson, they were taken through the Lesson Manual step by step in class so they were clear about the lesson objectives, the task, the resources available to them to complete the task and the associated assessment. The coordinator also went through with the students the background story on “soup dumplings” which was written in Chinese and included in the Lesson Manual. This enabled the teachers to highlight and explain any new vocabulary (minimal) and discuss the cultural content (particularly with reference to any real life experiences students may have had with “soup dumplings”). It also served the purpose of getting students into the mental mode required for reading and comprehending characters and primed them on vocabulary, expressions and concepts that would be required during the process of completing the set task.

In the first assignment task, the food ordering scenario, students were organised into groups of four (although some had up to six for various reasons). The lesson was specifically designed to promote group collaboration with the hope that three kinds of informal mentoring might take place. These were: mentoring with technical skills (some students were more experienced with virtual worlds, while some were more experienced with skills required to set up and input Chinese characters on a computer); mentoring with regard to task requirements (some students grasped the task requirements better for a range of reasons); and mentoring in relation to language knowledge/skills (some students more confident/better prepared).

In the second assignment task, the dumpling scenario, students were not formally organised into groups and the task could be completed individually. However, they were encouraged to work in pairs to provide mutual support during the task, and many students did indeed work with other peers. The assessment milestones throughout the lesson (finding the teacher’s friend, finding the vegetable market, buying the missing ingredient, making the dumplings in the kitchen) were marked on an individual basis.

In both assignments students received immediate feedback on language output that they produced. In the restaurant scenario when ordering food, if incorrect terminology was used for ordering or for the name of a dish, students were asked by the teacher playing the part of the restaurant staff for clarification via a direct question or via a “recasting” of the incorrect usage with correct usage. In some cases, students corrected others in their group.

In the dumpling scenario students received three main kinds of feedback. First, when interacting with the automated “non-player characters” (NPC), if a student used language that was lexically or grammatically (less common) incorrect, the NPC asked for the utterance to be repeated. Often repetition led to a correction of form or lexis and the conversation moved on. Sometimes, however, due to the inputting of incorrect characters or terms, the conversation was unable to move on and the student became “stuck”. This often led to the teacher being called for help. At this point, the student received individualised (non-textual) feedback from one of the teaching staff to (a) isolate the problem and (b) to work on a solution. This usually resulted in the student being able to move forward with the task at hand. The third kind of feedback came from peers, generally when assistance was sought by a student unable to obtain information they required from conversation with an NPC.

 **Marking the assignment**
Prior to undertaking the tasks, all students and staff were issued with lesson goals, tasks and assessment guidelines.

A rubric was used in the second assessable lesson: 5% - Completed task, dumplings shown to teacher (must show dumplings to University teacher in SL). 4% - Did not complete task, but successfully purchased necessary item for completion of task (must show item to University teacher in SL). 3% - Did not complete task, did not successfully purchase necessary item, but found location where item is sold (must show instructions for getting to location to University teacher in SL). 1% - Did not complete task, did not purchase necessary item, did not find location where item is sold, did find Teacher Z’s friend (must show dialogue where friend confirms they know Teacher Z to University teacher in SL).

The assignment was new and innovative, therefore hard to measure against any benchmarks.

 **Communicating the results**
Students received feedback by means of completion/non-completion of the set tasks. Some students pointed out that the assignment was not equitable to everyone – some groups didn’t get to the point of dishes on table. They didn't have enough time to finish, so the co-ordinator went back to the logs to see what they were planning to order. He explained this to them and said he would ensure there would be equity.

 **Evaluating and improving the assignment**
To evaluate the assignment, the co-ordinator conferred with another staff member (from the Education Faculty) who was collaborating with him on a self-efficacy study, and was auditing the sessions. He noted:

"Some consideration was given during the second assessable lesson design process to make photographs taken by students during the lesson part of the assessment for the lesson (e.g. photographs that identify particular food items that students were asked to locate in the vegetable market); however it was eventually decided that for the sake of simplicity and to prevent any unnecessary stress, students would not be required to take, store and transmit photographs. In future lessons, as the level of students’ technical skills operating the interface become more advanced it may be possible to work this kind of assessment into the lesson design."

On an interesting note, one student who was not able to be physically present in the computer lab during the second assessable lesson due to being interstate and who completed the lesson in Second Life from her interstate location did actually take photographs (screen grabs) as evidence of her having completed various stages of the assessment.

Reflecting on the assignment, teaching staff felt that it was not as easy to manage as an online quiz, but more straightforward than an essay. The lecturer noted that in the future aspects of this lesson would have to be re-designed to ensure that the set assessment tasks can be completed in the allotted time. The assignment involved a quiz that students were to complete in Moodle after the first assessment task. However, only some students completed all or part of the quiz. The quiz provided students with automatic correction and feedback. Students also received feedback about their work throughout the lesson as the teaching staff went around and assisted with and commented on various issues. However, the first assignment task proved to be difficult to manage in the computer lab. There was a lot going on, and it was difficult to ensure all students received useful feedback. The lecturer noted that all students would have received useful feedback if they’d had time to do the quiz. He wondered to what extent the assignment in fact measured students' initiative, rather than their learning.

With regard to making changes to the assignment in the future, the lecturer thought that "Maybe students could be encouraged to take on an Asian identity in future – it may be that if students were required to take on a persona that could, visually at least, be closely identified with the target language /culture it may, for some students, allow them to relate better to or differently to the virtual cultural and linguistic environment. This in turn may have some impact on learning outcomes through greater identification with the language / culture being learnt."

The assignment appeared to work well as an opportunity for students to use knowledge and skills while still in the classroom. The interaction within Second Life occurred in two modes. The first was the visual mode, with all participants able to see what all other participants were doing. This gave reasonably clear visual clues about what stage of a particular task various groups/individuals were at. The second was text-based dialogue (in English and Chinese). Particularly during the first of the two assessed lessons, the food ordering scenario, students communicated via group instant messaging (IM) in the form of text-based discussion. All discussion in this mode was visible to all members of a particular group, although not visible to members of other groups. In addition, all text-based discussion in all groups was visible via the “‘group IM” to the teaching staff involved. All student work was only accessible for the duration of the lesson. However, it would be possible to make the text-based and visual representations of work available to all students via posting on a commonly accessible website (this was not done for beginner lessons, but has been done in the past for more advanced learner work).

Interaction "around the computer monitor" was also actively encouraged, with learners actively seeking help from peers who would often walk around the computer island to look at the screen of the person they were helping. This form of “access” was, again, only available for the duration of the lesson.

In a sense then, this type of assessment (i.e. of communicative competence) was engaging students in striving for "communicative excellence” rather than a narrowly defined linguistic excellence. This kind of assessment probably could be done in other formats, but again logistically it would possibly be more difficult, and certainly harder to contextualise.

In other Second Life learning activities, making objects can raise an issue of Intellectual Property rights. The content for the lessons concerned was only public for a very limited period of time, i.e. the duration of the lesson. Once each group finished their lesson, the resulting content - that is the food ordered or the dish that was made in the kitchen - was removed after being noted down. Content in the form of logged dialogue, while more permanent, has not been and will not be at this stage made public. Photographs taken by staff during the session at this stage will also not be made public, although there is the potential for this to be done in future lessons.

 **Selected documents from this case study**
Tea house lesson instructions: Tea House Lesson Instructions Pt2.pdf

Screen shot of Second Life island set up for Chinese lessons: Second_Life_island_for_Chinese_class.PNG

Screen shot showing student avatars during the restaurant lesson: Screen_shot_of_restaurant_lesson_VW.PNG

Second screen shot showing student avatars during the restaurant lesson: Screen_shot_of_restaurant_lesson_VW2.PNG

 Page source: <span style="font-family: Arial,Helvetica,sans-serif;"> Gray, K., Waycott, J., Thompson, C., Clerehan, R., Sheard, J., Hamilton, M., & Richardson, J. (2011) //Using Social Web (Web 2.0) Activities for Student Assessment: Resources for University Learning and Teaching//. Retrieved from https://web2assessmentresources.wikispaces.com