How do students take notes in VR language lessons? Or should they?

Note-taking is good for learning (Friedman, n.d.). We learned how to do it at school and university, and we teach it to our students. In language lessons, teachers write up language and grammar they want to teach on the board, projected screen, or chat box, and students copy these if they wish. Students also take notes from reading, during discussions, or when preparing for role-plays. Many other activities ask for students to note down something. But in fully immersive VR, taking notes while wearing a head-mounted display is currently rather difficult and disrupts the experience.

While developers are working on technical solutions to allow taking notes in fully immersive VR, I would like to look at the issue from a different perspective and ask: 

  1. Is there value in not taking notes during an immersive VR language lesson?
  2. Could note-taking in VR look different from how we are used to doing it?

Is note-taking always beneficial?

There are, in fact, situations and environments where taking notes during a lesson can be counterproductive.

Note-taking, even in one’s first language, demands a lot of cognitive effort as it requires comprehension, writing and learning simultaneously (Piolat et al., 2005), or as Hong (2017:19) puts it, students have to ‘balance both information intake and synthesized information production’, unless they just write down everything verbatim without understanding or thinking, in which case it is distracting them from paying attention without deriving any benefit from it (Friedman, n.d; Hong, 2017). If this is the case in one’s first language, imagine the cognitive challenge for a language learner.

There is also research (Lin and Bigenho, 2011) that suggests that when the environment is highly distracting, for example there is visual and auditory input at the same time, it is better to focus on the task and not to take notes. 

VR is a much more complex multimodal environment than two simultaneous input modes. Furthermore, the learner in a live VR language lesson is not passively taking in input (we hope anyway), but is interacting with the environment, objects and other people. A live, fully immersive VR lesson is a highly interactive learning experience, where students actively do things, such as role-play, visit places together and collaboratively solve problems. Stopping to take notes would interrupt this experience and distract both the note-taker and the others involved in the same experience.

Why note-taking during language lessons in VR might not always be necessary?

There is a lot of research (Cai, Tay and Ngo, 2013 – Chapter 1; Lan, 2015; Greenwald, 2018; Legault et al., 2019) that shows that experiential, interactive learning in VR helps comprehension and retention. In VR students have experiences, e.g. they go shopping, go on excursions, tell stories around a campfire, have a birthday party in the backyard, go through check-in at the airport and have business negotiations sitting in an office. We don’t always remember passive input from lectures, but we do remember events and experiences without taking notes. 

In fully immersive VR, spatial awareness is enhanced, which improves memory recall ability (Krokos et al. 2018), and spatial presence and motivation affects memory retention in language learning positively (Cho, 2018). 

Embodied learning research shows that movements and gestures help learning, including language learning (Wallong and Lindgren, 2017, in Spector, Lockee and Childress;  Macedonia, Lehner & Repetto, 2020; Andrä et al., 2020; Fuhrman et al., 2021). In VR students move and use their physical body to gesture, and their avatars move in the 3D space (and if they want, their physical bodies too). 

Looking at the above research, we can conclude that students might not need the ‘crutch’ of note-taking in immersive VR in order to remember the language presented and used in a session. This would make for fascinating research!

Although some teachers will attempt to teach a whole lesson or course in VR, I strongly believe that a blended format is much more suitable. A well-designed blended course is in many ways superior to a course delivered in only one mode. In a blended or flipped course, the VR session will be one of several stages in a lesson. The main input sessions and delayed and detailed feedback sessions, in which students do need and should take notes, will take place before or after the experiential, active VR session.

Finally, some students who struggle with writing in a regular class, for example, students with dyslexia or dyspraxia, can find it liberating not to have to take notes in a lesson. Why not give these students a break for once and take off the pressure in one stage of the lesson by using VR, where they can focus on their strength?

What if you do want students to have notes from a lesson?

There might still be situations or lessons where you do feel notes would be useful or will be needed for a later activity. What can you do within the possibilities of the current state of VR technology? 

The teacher takes notes

Teachers have always elicited language from students and written up incidental language that comes up in a lesson on the board or the projected screen to review at the end of the lesson, or for students to copy. For this purpose (and to provide them with other facilitation tools), Immerse developed a desktop VR app for teachers. This allows them to use their computer keyboard to type text on whiteboards in the VR environment. Students can take screenshots of these, or the teacher can copy the text and share them with the students after the lesson.

Placable whiteboards in the Immerse app

If there is no desktop VR app for the teacher, or for longer or formatted text, they will have to prepare slides with the language and grammar points they want to teach beforehand, and display these in-world. 

Whiteboard with post-its and a multimedia board in the Immerse app

Depending on the in-app writing tools available, teachers can also hand write and draw incidental vocabulary or explanations on whiteboards, other surfaces or in ‘the air’ in the 3D environment, but currently, this is not suitable for longer texts.

We also want to give agency to the students, so let’s look at other options.

Pair students up

Version one: VR + non-VR

In each pair, one student puts on the VR headset and is in the VR environment, while their partner sits next to them and asks questions, listens and notes down the answers or descriptions the VR student provides. Then they can swap. Afterwards, they can use their notes for follow-up activities. This works in multi-user VR environments as well as with single user experiences (e.g. 360° videos). There is an example of this kind of setup with all the details of the activity in this study by Lin et al. (2021), and a short video of the same activity below:

This is something that is often done in a class where there aren’t enough headsets for all students. However, it is also a pedagogically valuable setup that can make many tasks more communicative and collaborative. This works best in the face-to-face classroom. Online, you could put each pair in a different breakout room, but it would mean that one student would have to be both in VR and in the Zoom session, which might cause technical issues, e.g. with sound.

Version two: VR headset + desktop VR

In this version, both students are in the VR environment together – one with their VR headset, the other using the desktop version of the app. The desktop VR student takes notes.

Record sessions and take screenshots

VR might not allow for efficient writing yet, but it is easy to take screenshots and record all or part of a session, some apps even have a time travel option in which learners can revisit or re-experience the whole lesson in 3D, as described in Hong (2017:27), making use of VR’s unique ‘spatial, temporal and sensory capacity.’ Recordings can be combined with hyperlinked screenshots, where the recording serves as verbatim notes and screenshots like summaries of key events or moments (Hong, 2017; Greenwald, 2018 and Greenwald et al., 2016). The VR game Myst makes use of screenshots for their journaling feature. 

This is a good solution and has a lot of pedagogical value in itself if integrated well into lessons. Learners will neither have the time nor motivation to watch or re-experience each session to take notes. They will also not automatically know how to make best use of this great tool. They need to be given meaningful tasks and clear instructions, so they know what to do and why they are doing it, and have motivation to do it. We know, for example, that blended learning works best when online and offline or in-class and out-of-class activities are integrated, and when what the students work through on their own is used in class, and forms part of their grade. We need to do the same with VR recordings.

One task could be to watch a recording of themselves (or a classmate) performing a role-play, discussion or presentation activity and note down what was good (good language use, intonation, interaction…) and what was not so successful, and how they could improve it, by looking at the language or grammar handouts, or rubrics, they were provided with. This is a much better solution than providing instant feedback within the live VR lesson. Instant feedback, even in a traditional classroom setting, often doesn’t work well as students are still in the experience and performing, and we would either have to interrupt them, and distract them from the experience. Even if we provide feedback immediately after the performance, students will often not remember what they said, so can’t connect the feedback with their performance. 

Being able to watch a recording of their performance allows them to see and hear themselves, and they can then notice their errors and note these down, and then work on improving their language. This can be more effective than the teacher simply providing all the feedback, it also feels more empowering.

A follow-up task could be to collaborate with another classmate, transcribe their part of a role-play or discussion, rewrite the dialogue and send it to the teacher for evaluation and further feedback, or collect all dialogues in one place so everyone can read the others’ versions. Transcription tasks raise students’ language awareness and help them focus on form (Stillwell et al., 2009) and students find them generally useful, and interesting (Lynch, 2001; Stillwell et al. 2009), which can increase motivation to complete the task.

Screenshots from different stages of a lesson can also be helpful in reminding students what happened in a session or to keep record of language written up or displayed by the teacher. They are a good alternative to recording a whole session as video, take up less space and are easier to browse through. The downside, of course, is that students can’t hear the language again. But students could be asked to use screenshots of their role-plays or other communication tasks to, for example, create a comic strip, half remembering and half inventing the dialogues.

Students can take snapshots themselves of text that is displayed or anything else they want to remember. They can also record activities themselves, which can be part of a task. However, when they are performing, either the teacher needs to do this or assign it to another non-performing student, unless the recording and screenshot-taking is automated (Hong, 2017).

What are the limitations of these solutions? Would these solutions work for any class and subject?

These solutions would work with courses that use active, communicative, task-based, experiential approaches rather than lecture-style lessons, or reading- or grammar-heavy lessons. I have seen technical solutions for listening to lectures in VR and taking notes with a physical keyboard or even a graphic tablet. But just because something is technically possible, it doesn’t mean it makes pedagogical sense. We don’t need to shoehorn every learning task into VR just because we can; instead, we should use it for its strengths, for where it adds value. 

Recording is certainly an option, however, if note-taking or writing skills development is an essential part of a course, then don’t use VR, or use a blended solution. For example, if listening and note-taking is one of the key skills being taught, or students have to take part in a more comprehensive seminar discussion in which they need to make notes of what points others are making, or their own ideas for responding, VR is not yet suitable, and might never be the best option. Almost all courses benefit from a blended approach, no matter how developed the technology is, because it allows students with diverse needs to do some tasks that need more thinking time and cognitive effort in their own time and at their own pace, rather than cramming everything into a live lesson.

Conclusion

With new technology we often have the tendency to want to do the same things in the same way as before. And if this isn’t possible, we see it as a challenge or downside of that tool or environment. But then, with time and experience, we find out what the technical and pedagogical affordances of the new technology are, and can use those to our and our learners’ benefit. Better note-taking and writing in general in fully immersive VR will soon be possible, but even then, we need to think about whether we can and should do things differently with this immersive technology – for example, ‘note-taking reimagined in a virtual space’ (Hong, 2017). Perhaps we can find out ways of doing and learning in VR which might in some cases be superior to how we have been doing things before.

Read part two on note-taking in virtual reality for practical suggestions.

Share…